# Voice agents with Realtime Video — Sidney Primas, LemonSlice

## Executive summary

LemonSlice aims to break the Avatar Turing test by creating highly realistic, real-time video avatars. The core technical challenges addressed include achieving emotional expressiveness (requiring specialized audio encoders beyond monotone audiobook data), mitigating error accumulation over extended generation periods (e.g., 8+ hours), and optimizing for real-time performance. A major focus is on the 'model harness'—the complex orchestration of threads and queues across GPU/CPU to ensure uninterrupted, stutter-free video streaming at scale. The company also highlights cost parity, noting that generating high-resolution video costs comparably to running a voice model.

## Key takeaways

- Real-Time Video Generation Challenges: Generating real-time avatars requires training models with an attention mask that enforces looking only into the past, as future frames do not exist yet. Furthermore, speeding up generation involves collapsing many denoising steps (e.g., 30 steps) down to a single step.
- Error Accumulation Mitigation: A significant technical hurdle is error accumulation, where errors introduced in previous video blocks compound over time. The company claims to have developed a novel method to generate very long videos with no noticeable error buildup.
- Model Harness and Cost Parity: The most durable value lies in the 'model harness'—the orchestration of threads and queues across GPU and CPU to maintain real-time, stutter-free video. Surprisingly, serving this complex visual layer costs about the same as serving a voice model.
- Future Direction: Emotional Engine: The next generation involves building an 'emotion engine' that predicts and controls emotional reactions and actions based on both audio input and text input, moving beyond current awkward interactions.

## Technical details

- World Model Architecture: The approach involves taking general 'world models' and focusing them specifically on human subjects. This allows for emergent properties like full body movement, object interaction, micro-expressions, and physics understanding.
- Audio Encoding for Expressiveness: Achieving accurate emotions and facial expressions requires specialized audio encoders because standard encoders are often trained on monotone audiobooks. The video model must be trained with high-quality, expressive audio embeddings.
- Model Optimization: The system uses a Video DIT model and requires significant optimization to move from multi-step denoising (e.g., 30 steps) down to single-step generation for real-time performance.
- End-to-End EQ Layer: The long-term vision is a single, end-to-end 'EQ layer' model that takes direct user video and audio input (like a FaceTime call) to produce avatar video and audio output internally managing emotional state.

## Practical implications

- The technology can serve as a visual layer on top of existing LLMs and voice agents for applications like language learning or AI sales calls.
- Low cost parity enables consumer use cases, making high-resolution video generation accessible.

## Topics

AI Avatars, World Models, Real-Time Video Generation, Machine Learning Optimization, Emotional AI, LemonSlice

Source: https://www.youtube.com/watch?v=z1dqv74SpUs