While my guitar gently speaks — Todd Fisher, Philo Ventures
Summary
Todd Fisher details his project to make a guitar speak, showcasing an evolution from simple audio effects to complex AI-driven speech and singing synthesis. The process involves integrating multiple technologies: using JUCE for plugin development, employing advanced signal processing techniques like Energy Gap Segmentation and Sonority Peak Syllabification for word slicing, and utilizing algorithms such as YIN pitch detection and vocoders to achieve synthesized vocals.
Key takeaways
-
Initial Goal: Making the Guitar Speak
The project started with the goal of having a guitar respond to questions from the room, moving beyond traditional effects like the talk box. This was initially achieved by playing pre-recorded audio files.
-
Word Segmentation Challenges
12:30
Automatically slicing speech into words is challenging because standard methods (like Energy Gap Segmentation, which relies on silence) fail when spoken words run together. The speaker explored Sonority Peak Syllabification as an alternative.
-
Singing Synthesis Pipeline
17:03
To make the guitar sing, the process requires detecting the fundamental frequency using the YIN algorithm. This pitch is then used to shape a synthesized note (via ADSR) and passed through a vocoder, which maps it onto pre-baked, pitch-shifted vocal samples.
-
Advanced Interaction: LLM Integration
The most advanced step involves using speech-to-text (Whisper) to capture a question, feeding it into a local Large Language Model (LLM), and then playing the resulting text response through the guitar's synthesis system.
Technical details
-
Audio Framework & Development
606s
The project utilizes JUCE as a framework for building audio software plugins that integrate into Digital Audio Workstations (DAWs) like Logic Pro. The DAW is described as the 'IDE but for musicians.'
-
Speech Synthesis & Segmentation
750s
Initial text-to-speech was handled using Piper and built-in Apple tools. To segment speech into words, the speaker investigated Energy Gap Segmentation (which fails without silence) and Sonority Peak Syllabification (identifying vowels to locate syllables). Manual editing of segments was required for optimal results.
-
Pitch Detection & Synthesis
1023s
For singing, the YIN pitch algorithm is used to detect the fundamental frequency from a fretted note. This detected pitch is then shaped into a synthesized sawtooth/note (using ADSR) and passed through a vocoder, which effectively 'fills up the cavity' of the voice.
-
AI Integration Pipeline
The full conversational loop involves: Microphone input $\rightarrow$ Speech-to-Text (Whisper) $\rightarrow$ Raw Text Output $\rightarrow$ Local LLM Processing $\rightarrow$ Synthesized Audio Output on Guitar.
Mentioned resources
- JUCE
- Piper
- Whisper
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.