# While my guitar gently speaks — Todd Fisher, Philo Ventures

## Executive summary

Todd Fisher details his project to make a guitar speak, showcasing an evolution from simple audio effects to complex AI-driven speech and singing synthesis. The process involves integrating multiple technologies: using JUCE for plugin development, employing advanced signal processing techniques like Energy Gap Segmentation and Sonority Peak Syllabification for word slicing, and utilizing algorithms such as YIN pitch detection and vocoders to achieve synthesized vocals.

## Key takeaways

- Initial Goal: Making the Guitar Speak: The project started with the goal of having a guitar respond to questions from the room, moving beyond traditional effects like the talk box. This was initially achieved by playing pre-recorded audio files.
- Word Segmentation Challenges: Automatically slicing speech into words is challenging because standard methods (like Energy Gap Segmentation, which relies on silence) fail when spoken words run together. The speaker explored Sonority Peak Syllabification as an alternative.
- Singing Synthesis Pipeline: To make the guitar sing, the process requires detecting the fundamental frequency using the YIN algorithm. This pitch is then used to shape a synthesized note (via ADSR) and passed through a vocoder, which maps it onto pre-baked, pitch-shifted vocal samples.
- Advanced Interaction: LLM Integration: The most advanced step involves using speech-to-text (Whisper) to capture a question, feeding it into a local Large Language Model (LLM), and then playing the resulting text response through the guitar's synthesis system.

## Technical details

- Audio Framework & Development: The project utilizes JUCE as a framework for building audio software plugins that integrate into Digital Audio Workstations (DAWs) like Logic Pro. The DAW is described as the 'IDE but for musicians.'
- Speech Synthesis & Segmentation: Initial text-to-speech was handled using Piper and built-in Apple tools. To segment speech into words, the speaker investigated Energy Gap Segmentation (which fails without silence) and Sonority Peak Syllabification (identifying vowels to locate syllables). Manual editing of segments was required for optimal results.
- Pitch Detection & Synthesis: For singing, the YIN pitch algorithm is used to detect the fundamental frequency from a fretted note. This detected pitch is then shaped into a synthesized sawtooth/note (using ADSR) and passed through a vocoder, which effectively 'fills up the cavity' of the voice.
- AI Integration Pipeline: The full conversational loop involves: Microphone input $\rightarrow$ Speech-to-Text (Whisper) $\rightarrow$ Raw Text Output $\rightarrow$ Local LLM Processing $\rightarrow$ Synthesized Audio Output on Guitar.

## Practical implications

- The talk emphasizes that modern AI tools significantly lower the barrier to entry for building complex passion projects and side projects.
- Engineers are encouraged to leverage these technologies (especially AI) to build personal projects rather than waiting for formal assignments.

## Topics

Digital Signal Processing (DSP), Generative AI, Audio Synthesis, Machine Learning, Embedded Systems/Plugins, JUCE, Piper, Whisper

Source: https://www.youtube.com/watch?v=E_Txocq-Lrw