While my guitar gently speaks — Todd Fisher, Philo Ventures
Todd Fisher details his project to make a guitar speak, showcasing an evolution from simple audio effects to complex AI-driven speech and singing synthesis. The process involves integrating multiple technologies: using JUCE for plugin development, employing advanced signal processing techniques like Energy Gap Segmentation and Sonority Peak Syllabification for word slicing, and utilizing algorithms such as YIN pitch detection and vocoders to achieve synthesized vocals.
Key takeaways
-
Initial Goal: Making the Guitar Speak
The project started with the goal of having a guitar respond to questions from the room, moving beyond traditional effects like the talk box. This was initially achieved by playing pre-recorded audio files.
-
Word Segmentation Challenges
12:30
Automatically slicing speech into words is challenging because standard methods (like Energy Gap Segmentation, which relies on silence) fail when spoken words run together. The speaker explored Sonority Peak Syllabification as an alternative.
-
Singing Synthesis Pipeline
17:03
To make the guitar sing, the process requires detecting the fundamental frequency using the YIN algorithm. This pitch is then used to shape a synthesized note (via ADSR) and passed through a vocoder, which maps it onto pre-baked, pitch-shifted vocal samples.
-
Advanced Interaction: LLM Integration
The most advanced step involves using speech-to-text (Whisper) to capture a question, feeding it into a local Large Language Model (LLM), and then playing the resulting text response through the guitar's synthesis system.