# How to build voice agent with AssemblyAI TTS & Livekit

## Executive summary

This guide demonstrates how to build a sophisticated voice agent by seamlessly integrating three core AI components: Speech-to-Text (STT), Large Language Model (LLM) processing, and Text-to-Speech (TTS). The process is streamlined using AssemblyAI's models and LiveKit, allowing developers to build complex agents—including those with tool calling capabilities—with minimal setup. The final steps cover advanced features like cloning custom voices and connecting the agent to a phone number.

## Key takeaways

- Three Pillars of a Voice Agent: A voice agent requires three components: 1) Speech-to-Text (STT) for the 'ears,' 2) an LLM (the 'brain') for logic, and 3) Text-to-Speech (TTS) for the 'voice.' AssemblyAI provides seamless models for all three.
- Simplified Agent Setup: The setup process is simplified by using a reference repository (AssemblyAI/LiveKit) and providing the AssemblyAI API key, allowing the framework to automatically configure STT, LLM, and TTS.
- Implementing Tool Calling: LLMs can be enhanced with tool calling (e.g., `get_weather`). When the agent detects a need for external data (like weather), it can automatically call a defined function, making the agent smarter and more functional.
- Custom Voice Cloning: The TTS model supports voice cloning. Users can clone a voice using as little as 20 seconds of audio and can access over 4,000 voices across multiple accents and languages.

## Technical details

- Voice Agent Architecture: The system relies on three integrated models: AssemblyAI's Speech-to-Text model, an LLM gateway (e.g., GPT-3.5 or a default OpenAI model), and AssemblyAI's Text-to-Speech model.
- LiveKit Integration: LiveKit is used as the open-source voice agent provider. The process involves cloning the reference repository, installing the LiveKit CLI, and providing the AssemblyAI API key to initialize the agent.
- Agent Customization: Developers can customize the agent by defining system prompts, setting specific keywords (e.g., names, medical terms) to improve recognition, and defining custom instructions (e.g., what to say when the customer says 'goodbye').
- Tool Calling Mechanism: Tool calling allows the LLM to detect when external information is needed (e.g., weather) and automatically execute a defined function (e.g., `get_weather`) using an API service.
- Deployment and Calling: LiveKit provides a service to connect the voice agent to a phone number, enabling the agent to make calls on the user's behalf.

## Practical implications

- The integration of STT, LLM, and TTS into a single, modular pipeline significantly lowers the barrier to entry for building complex voice applications.
- Tool calling capability allows voice agents to move beyond simple conversational flow and perform real-world actions (e.g., checking live data, booking appointments).
- Voice cloning capability allows for highly personalized and branded customer service experiences.

## Topics

Voice Agents, Speech Recognition, LLMs, Text-to-Speech, LiveKit, Tool Calling, AI Development, AssemblyAI Text-to-Speech Model, AssemblyAI/assemblyai-livekit-reference-voice-agent, AssemblyAI Dashboard

Source: https://www.youtube.com/watch?v=PF1f4GtOYaw