The Video Signal

Channel

AssemblyAI

Digests from AssemblyAI — AssemblyAI videos on speech AI, applied AI, and developer workflows.

Workshop: Building and optimizing dictation features thumbnail

· 36:06

Workshop: Building and optimizing dictation features

This workshop details the architecture for building low-latency dictation features using AssemblyAI's APIs. The core strategy involves leveraging the Sync API—a single POST request—to achieve speed superior to traditional async or streaming methods. The process is optimized through three key stages: using key terms prompting for accuracy, implementing connection warming to minimize network overhead (DNS, TCP, TLS), and running a cleanup pass via LLM Gateway (e.g., Qwen3.5 4B Fast) to refine raw transcripts into polished, intent-preserving text. The goal is to deliver results on screen in under one second.

Key takeaways

  1. Sync API Preference 5:35

    The Sync API is preferred for dictation because it avoids the overhead of maintaining a WebSocket connection (streaming) and bypasses the inherent latency floor of traditional async endpoints, which is unsuitable for short, burst dictation sessions.

  2. Latency Optimization via Warming 11:40

    To minimize latency, it is critical to call a warm endpoint (`client.sync.warm`) while the user is recording. This pre-pays the networking costs (DNS, TCP, TLS handshake), ensuring the subsequent transcription request goes straight to inference.

  3. Cleanup Pass with LLM Gateway 18:20

    A cleanup step using an LLM (like Qwen3.5 4B Fast) is necessary to transform raw, spoken text (e.g., 'I think we should meet in 5 minutes') into polished, corrected text while preserving the original intent. This is achieved by prompting the model to act as a function in a pipeline, not an assistant.

  4. Upcoming Dictation API 30:00

    AssemblyAI is rolling out a dedicated Dictation API (`client.dictation.describe`) that will wrap the entire loop (STT + Cleanup) into a single, simplified call, making the process easier for developers.

Watch on YouTube Full article

Build Smarter Voice Agents thumbnail

· 46:14

Build Smarter Voice Agents

This panel discussion provides deep insights into building production-grade voice AI agents. Key architectural recommendations favor the deconstructed cascading pipeline (ASR $\rightarrow$ LLM $\rightarrow$ TTS) due to its superior flexibility for optimization and model swapping. Engineers must prioritize managing latency within a 1–1.5 second budget, implementing robust fallback systems across all stack components (ASR, LLM, TTS), and utilizing advanced context management techniques like 'Scratchpads' to maintain conversational continuity over long interactions.

Key takeaways

  1. Architectural Choice: Cascading Pipeline 1:45

    The cascading architecture is preferred because it allows for individual optimization of the ASR, LLM, and TTS layers. This modularity provides greater flexibility than a full Speech-to-Speech (S2S) stack when integrating new models or optimizing specific components.

  2. Latency Management 4:23

    The 'golden metric' for voice agent response time is between 1 to 1.5 seconds. Exceeding this budget can be unnerving for users, making latency a primary design constraint over pure accuracy in many cases.

  3. Context and Memory Management 18:05

    To prevent negative user sentiment from repeating information, agents must implement context stores (e.g., 'Scratchpads') to track customer profiles, preferences, and key facts across multiple turns or sessions.

Watch on YouTube Full article

Agent Context Carryover in Livekit Tutorial thumbnail

· 8:07

Agent Context Carryover in Livekit Tutorial

This tutorial demonstrates 'Agent Context Carryover,' a feature released for LiveKit using AssemblyAI's Universal 3.5 Pro model. This capability automatically feeds conversational context into the speech-to-text model, significantly boosting transcription accuracy—especially for proper nouns and key terms—without requiring manual plumbing or explicit context prompting from the developer.

Key takeaways

  1. Agent Context Carryover Functionality

    The feature automatically provides conversational context to the model, improving transcription accuracy when building voice agents on LiveKit. This is achieved by enabling a single parameter rather than implementing complex key term handling.

  2. Implementation Simplicity

    AssemblyAI's LiveKit plugin handles the necessary context plumbing automatically, allowing developers to gain accuracy benefits without setting up custom logic for key terms or sending agent messages manually.

Watch on YouTube Full article

Universal 3.5 Pro Demo: Smarter Speech-to-Text with Contextual Awareness thumbnail

· 10:07

Universal 3.5 Pro Demo: Smarter Speech-to-Text with Contextual Awareness

This demo introduces Universal 3.5 Pro, an advanced Speech-to-Text (STT) model designed to significantly boost transcription accuracy through enhanced contextual awareness. Key features include passing domain-specific prompts (e.g., 'cardiology consultation'), applying context to key terms to prevent misapplication, and supporting dynamic mid-call prompt updates via API calls. Furthermore, the model retains conversation history (agent context), allowing it to accurately transcribe user input even in poor audio conditions by understanding the situational flow of a voice agent interaction.

Key takeaways

  1. Contextual Prompting for Domain Accuracy

    Passing detailed information about the audio content (e.g., 'cardiology consultation between Dr. Smith and elderly patient regarding chest pain...') dramatically improves model accuracy within specific domains. The more specific the prompt, the better the results.

  2. Contextual Key Terms 2:00

    Unlike previous methods where key terms were applied blindly, Universal 3.5 Pro allows users to define what a key term represents (e.g., 'The user's name is Zachary Klebanoff'). This prevents the model from incorrectly applying terminology based solely on acoustic similarity.

  3. Dynamic Mid-Call Prompt Updates 2:55

    The prompt can be updated in real time via the API (not available in the playground demo). This is crucial for voice agents, allowing tool calls or external data to adjust the model's context mid-conversation.

  4. Conversation/Agent Context 3:30

    The model retains previous transcriptions and accepts LLM-generated responses from a voice agent as context. This provides situational awareness, improving accuracy even in poor audio conditions and reducing the Word Error Rate (WER) on voice agent datasets.

Watch on YouTube Full article