Customizing Vertical Voice Agents with ASR and LLM Fine-Tuning | Nemotron Labs
Summary
This session provides a deep dive into building sophisticated, domain-specific voice agents by customizing foundation models. Experts demonstrated how to fine-tune both Automatic Speech Recognition (ASR) models (e.g., Nemotron ASR) and Large Language Models (LLMs) to achieve high accuracy and low latency in specialized domains, such as air traffic control. The architecture emphasizes a low-latency central loop, utilizing WebRTC for real-time audio handling and advanced techniques to manage context and performance in production.
Key takeaways
-
Fine-Tuning for Domain Specificity
2:00
Fine-tuning ASR models on specialized vocabulary (e.g., air traffic control lingo) significantly improves recognition accuracy, especially in noisy environments. For LLMs, post-training is crucial for explaining complex, domain-specific terminology that base models struggle with.
-
Low-Latency Architecture Components
2:30
A typical voice agent architecture includes an audio input (via WebRTC), a fine-tuned ASR model for transcription, an LLM acting as the 'brain' to manage multi-turn context, and a speech generation output. Parallel turn detection processes (like using 'Jev as a judge') can be added to evaluate model performance.
-
Performance Metrics and Optimization
4:20
Achieving low latency is critical. The fine-tuned ASR model demonstrated final transcripts in less than 60 milliseconds, and the fine-tuned LLM achieved a time to first token in under 100 milliseconds, significantly outperforming general cloud-hosted models (which can take 1,200–1,500 milliseconds).
-
Data Strategy for Customization
8:00
The process of customization is described as an 'art.' While real human data is best, synthetic data generation is highly useful for augmenting training sets and covering edge cases. The speakers recommend starting with small tests (e.g., 10 hours) and scaling up to 100 hours for production, but always verifying the use case first (e.g., checking if word boosting or a dictionary is sufficient).
Technical details
-
WebRTC vs. WebSockets for Audio
390s
For real-time audio over the internet, WebRTC is preferred over WebSockets because it is designed to handle packet loss and variable network latency gracefully. WebRTC uses UDP, allowing selective packet dropping, which is necessary for real-time audio, whereas WebSockets force in-order packet delivery, which can cause latency accumulation.
-
Model Customization Techniques
560s
For LLMs, both Supervised Fine-Tuning (SFT) and Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA are available. However, for larger models (e.g., 30B), the speakers generally recommend full weights SFT for stability and performance.
-
Full Duplex Voice Agents
690s
A full duplex experience is defined as a highly responsive, bidirectional conversation where the user can interrupt the agent or provide back-channeling. This is achievable using components from specialized libraries (like Pipecat) that manage the entire modern state-of-the-art voice agent pipeline.
Mentioned resources
- Pipecat
- Nemotron ASR
- Neotron 3 Nano
- NVIDIA Skills Orchestration
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.