Workshop: Building and optimizing dictation features
This workshop details the architecture for building low-latency dictation features using AssemblyAI's APIs. The core strategy involves leveraging the Sync API—a single POST request—to achieve speed superior to traditional async or streaming methods. The process is optimized through three key stages: using key terms prompting for accuracy, implementing connection warming to minimize network overhead (DNS, TCP, TLS), and running a cleanup pass via LLM Gateway (e.g., Qwen3.5 4B Fast) to refine raw transcripts into polished, intent-preserving text. The goal is to deliver results on screen in under one second.
Key takeaways
-
Sync API Preference
5:35
The Sync API is preferred for dictation because it avoids the overhead of maintaining a WebSocket connection (streaming) and bypasses the inherent latency floor of traditional async endpoints, which is unsuitable for short, burst dictation sessions.
-
Latency Optimization via Warming
11:40
To minimize latency, it is critical to call a warm endpoint (`client.sync.warm`) while the user is recording. This pre-pays the networking costs (DNS, TCP, TLS handshake), ensuring the subsequent transcription request goes straight to inference.
-
Cleanup Pass with LLM Gateway
18:20
A cleanup step using an LLM (like Qwen3.5 4B Fast) is necessary to transform raw, spoken text (e.g., 'I think we should meet in 5 minutes') into polished, corrected text while preserving the original intent. This is achieved by prompting the model to act as a function in a pipeline, not an assistant.
-
Upcoming Dictation API
30:00
AssemblyAI is rolling out a dedicated Dictation API (`client.dictation.describe`) that will wrap the entire loop (STT + Cleanup) into a single, simplified call, making the process easier for developers.
