# Event Recap: Build Smarter Voice Agents - New York Edition

## Executive summary

This recap details the complexities of building and deploying production-grade voice AI agents across two distinct sectors: professional networking (Boardy) and regulated healthcare (Flagler Health). Key challenges discussed include maintaining conversational flow, establishing user trust, managing multi-party video meeting interactions, and ensuring subsecond latency for natural conversation. The discussion highlights the difference between highly structured, goal-oriented flows (healthcare) and highly conversational, relationship-driven interactions (networking).

## Key takeaways

- Design Flow Differences: Healthcare voice agents require highly structured, step-by-step flows with strict guardrails (e.g., collecting insurance info) to prevent medical advice or deviation. Conversely, networking agents are designed to handle highly conversational, open-ended interactions to facilitate connections.
- The Importance of Trust and Disclosure: Building user trust is critical. Speakers emphasized that being upfront and immediately disclosing that the user is speaking to an AI (e.g., 'I'm Sarah and AI') is essential to prevent user frustration and loss of trust.
- Technical Challenge: Multi-Party Meetings: Handling voice agents in multi-person video meetings (like Google Meet) is technically difficult. The primary challenge is determining when the agent should speak (turn-taking) to avoid false positives (randomly jumping in) or false negatives (failing to reply).
- Achieving Low Latency: To feel like a natural conversation, the system must achieve subsecond latency. This requires advanced architecture, such as preemptively generating the entire voice pipeline while the user is speaking.

## Technical details

- Guardrails and Flow Control: For regulated environments, flows must be regimented, often following a flowchart to ensure critical data (like insurance or procedure details) is collected without deviation. Guardrails are necessary to prevent the AI from giving medical advice.
- Agent Observability and Testing: Monitoring agent performance requires advanced observability tools to track both successful goal completion and failure points. Testing is complex, requiring simulation of edge cases and human review of transcripts to identify where the system 'thinks' a goal was met when it wasn't.
- Voice Pipeline Architecture: Achieving natural conversation requires managing turn-taking and latency. Solutions involve using tools like LiveKit and implementing a class of logic to decide when to speak, coupled with preemptive generation of the voice pipeline to ensure subsecond response times.
- Long-Term Memory and Context: While LLMs handle context well within a single conversation, maintaining memory across multiple channels (voice, email, text) over long periods requires revamping the underlying architecture to support long-term memory and conversation history.

## Practical implications

- Focus on building robust guardrails that enforce structured data collection in regulated fields.
- Prioritize low-latency architecture (subsecond response time) to move beyond the 'software tool' feel and achieve a 'teammate' feel.
- Implement comprehensive observability tools that track not just completion, but the quality and success of the interaction.
- Design the AI's persona to be upfront and transparent about its AI nature to build immediate user trust.

## Topics

Voice AI, Conversational AI, Build Engineering, LLMs, Guardrails, LiveKit, Machine Learning, AssemblyAI, Twilio, Boardy, Flagler Health

Source: https://www.youtube.com/watch?v=fS4jdWagHtY