Voice agents with Realtime Video — Sidney Primas, LemonSlice
LemonSlice aims to break the Avatar Turing test by creating highly realistic, real-time video avatars. The core technical challenges addressed include achieving emotional expressiveness (requiring specialized audio encoders beyond monotone audiobook data), mitigating error accumulation over extended generation periods (e.g., 8+ hours), and optimizing for real-time performance. A major focus is on the 'model harness'—the complex orchestration of threads and queues across GPU/CPU to ensure uninterrupted, stutter-free video streaming at scale. The company also highlights cost parity, noting that generating high-resolution video costs comparably to running a voice model.
Key takeaways
-
Real-Time Video Generation Challenges
17:05
Generating real-time avatars requires training models with an attention mask that enforces looking only into the past, as future frames do not exist yet. Furthermore, speeding up generation involves collapsing many denoising steps (e.g., 30 steps) down to a single step.
-
Error Accumulation Mitigation
20:22
A significant technical hurdle is error accumulation, where errors introduced in previous video blocks compound over time. The company claims to have developed a novel method to generate very long videos with no noticeable error buildup.
-
Model Harness and Cost Parity
23:54
The most durable value lies in the 'model harness'—the orchestration of threads and queues across GPU and CPU to maintain real-time, stutter-free video. Surprisingly, serving this complex visual layer costs about the same as serving a voice model.
-
Future Direction: Emotional Engine
The next generation involves building an 'emotion engine' that predicts and controls emotional reactions and actions based on both audio input and text input, moving beyond current awkward interactions.