# Customizing Vertical Voice Agents with ASR and LLM Fine-Tuning | Nemotron Labs

## Executive summary

This session provides a deep dive into building sophisticated, domain-specific voice agents by customizing foundation models. Experts demonstrated how to fine-tune both Automatic Speech Recognition (ASR) models (e.g., Nemotron ASR) and Large Language Models (LLMs) to achieve high accuracy and low latency in specialized domains, such as air traffic control. The architecture emphasizes a low-latency central loop, utilizing WebRTC for real-time audio handling and advanced techniques to manage context and performance in production.

## Key takeaways

- Fine-Tuning for Domain Specificity: Fine-tuning ASR models on specialized vocabulary (e.g., air traffic control lingo) significantly improves recognition accuracy, especially in noisy environments. For LLMs, post-training is crucial for explaining complex, domain-specific terminology that base models struggle with.
- Low-Latency Architecture Components: A typical voice agent architecture includes an audio input (via WebRTC), a fine-tuned ASR model for transcription, an LLM acting as the 'brain' to manage multi-turn context, and a speech generation output. Parallel turn detection processes (like using 'Jev as a judge') can be added to evaluate model performance.
- Performance Metrics and Optimization: Achieving low latency is critical. The fine-tuned ASR model demonstrated final transcripts in less than 60 milliseconds, and the fine-tuned LLM achieved a time to first token in under 100 milliseconds, significantly outperforming general cloud-hosted models (which can take 1,200–1,500 milliseconds).
- Data Strategy for Customization: The process of customization is described as an 'art.' While real human data is best, synthetic data generation is highly useful for augmenting training sets and covering edge cases. The speakers recommend starting with small tests (e.g., 10 hours) and scaling up to 100 hours for production, but always verifying the use case first (e.g., checking if word boosting or a dictionary is sufficient).

## Technical details

- WebRTC vs. WebSockets for Audio: For real-time audio over the internet, WebRTC is preferred over WebSockets because it is designed to handle packet loss and variable network latency gracefully. WebRTC uses UDP, allowing selective packet dropping, which is necessary for real-time audio, whereas WebSockets force in-order packet delivery, which can cause latency accumulation.
- Model Customization Techniques: For LLMs, both Supervised Fine-Tuning (SFT) and Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA are available. However, for larger models (e.g., 30B), the speakers generally recommend full weights SFT for stability and performance.
- Full Duplex Voice Agents: A full duplex experience is defined as a highly responsive, bidirectional conversation where the user can interrupt the agent or provide back-channeling. This is achievable using components from specialized libraries (like Pipecat) that manage the entire modern state-of-the-art voice agent pipeline.

## Practical implications

- When building voice agents, prioritize low-latency components and use WebRTC for all internet-based audio streams.
- Do not assume a single model can handle all use cases; combine specialized ASR models with customized LLMs to maximize performance in niche domains.
- Systematically test and manage model degradation (regression) by maintaining a mix of training data (e.g., mixing specialized ATC data with general English speech).
- For build engineers, focus on the asynchronous architecture pattern: keep the fast voice loop separate from slower, long-running processes (like RAG lookups) to prevent blocking and maintain responsiveness.

## Topics

Voice AI, Speech Recognition (ASR), Large Language Models (LLM), Low Latency Systems, Model Fine-Tuning, WebRTC, Pipecat, Nemotron ASR, Neotron 3 Nano, NVIDIA Skills Orchestration

Source: https://www.youtube.com/watch?v=F_glNmFP_5M