Event Recap: Build Smarter Voice Agents - New York Edition
Summary
This recap details the complexities of building and deploying production-grade voice AI agents across two distinct sectors: professional networking (Boardy) and regulated healthcare (Flagler Health). Key challenges discussed include maintaining conversational flow, establishing user trust, managing multi-party video meeting interactions, and ensuring subsecond latency for natural conversation. The discussion highlights the difference between highly structured, goal-oriented flows (healthcare) and highly conversational, relationship-driven interactions (networking).
Key takeaways
-
Design Flow Differences
10:20
Healthcare voice agents require highly structured, step-by-step flows with strict guardrails (e.g., collecting insurance info) to prevent medical advice or deviation. Conversely, networking agents are designed to handle highly conversational, open-ended interactions to facilitate connections.
-
The Importance of Trust and Disclosure
21:20
Building user trust is critical. Speakers emphasized that being upfront and immediately disclosing that the user is speaking to an AI (e.g., 'I'm Sarah and AI') is essential to prevent user frustration and loss of trust.
-
Technical Challenge: Multi-Party Meetings
24:10
Handling voice agents in multi-person video meetings (like Google Meet) is technically difficult. The primary challenge is determining when the agent should speak (turn-taking) to avoid false positives (randomly jumping in) or false negatives (failing to reply).
-
Achieving Low Latency
25:00
To feel like a natural conversation, the system must achieve subsecond latency. This requires advanced architecture, such as preemptively generating the entire voice pipeline while the user is speaking.
Technical details
-
Guardrails and Flow Control
700s
For regulated environments, flows must be regimented, often following a flowchart to ensure critical data (like insurance or procedure details) is collected without deviation. Guardrails are necessary to prevent the AI from giving medical advice.
-
Agent Observability and Testing
1050s
Monitoring agent performance requires advanced observability tools to track both successful goal completion and failure points. Testing is complex, requiring simulation of edge cases and human review of transcripts to identify where the system 'thinks' a goal was met when it wasn't.
-
Voice Pipeline Architecture
1500s
Achieving natural conversation requires managing turn-taking and latency. Solutions involve using tools like LiveKit and implementing a class of logic to decide when to speak, coupled with preemptive generation of the voice pipeline to ensure subsecond response times.
-
Long-Term Memory and Context
While LLMs handle context well within a single conversation, maintaining memory across multiple channels (voice, email, text) over long periods requires revamping the underlying architecture to support long-term memory and conversation history.
Mentioned resources
- LiveKit
- AssemblyAI
- Twilio
- Boardy
- Flagler Health
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.