Shipping AI to a Million Patients Without an A/B Test — Jared Joselowitz, Ufonia
Summary
The talk details how Ufonia built a comprehensive safety and evaluation stack for Dora, a conversational AI used in clinical post-op follow-ups. Because randomized A/B testing is unethical and illegal when dealing with patients, the system cannot rely on reactive rollbacks or standard model benchmarks. Instead, the approach shifts to rigorous simulation (the 'inner loop') using frameworks like Matrix, which employs simulated patients (PatBot) and an expert LLM judge (BevJudge). Safety is proven by optimizing prompts against a cost matrix (e.g., prioritizing sensitivity over overall accuracy) and utilizing automated prompt optimizers like Jeppa, ensuring the system ships evidence, not just a model.
Key takeaways
-
Safety Constraints in Healthcare AI
3:50
Standard software safety nets (A/B testing, rollbacks) fail when dealing with patients because randomizing into a worse variant is unethical and illegal; once a call is made, it cannot be undone. The model card's benchmark scores are insufficient defense at post-incident reviews.
-
The Necessity of Simulation
10:50
Since real-world testing (the 'outer loop') is too risky, the process must emulate high-reliability industries like self-driving cars. The simulation framework, Matrix, uses an LLM (PatBot) to play the patient against hazards written by clinicians.
-
Automated Hazard Detection
13:50
A second LLM, BevJudge, validates simulated dialogues. It is trained and validated against a corpus of 240 examples labeled by 10 clinicians from 10 specialties, achieving expert-level performance (e.g., F1 score of 0.96) with near-perfect sensitivity.
-
Optimizing Prompts via Cost Matrix
17:00
Instead of manual prompt engineering, the process uses optimizers like Jeppa (Genetic Pareto), which iteratively updates prompts based on a defined cost matrix. This allows optimization for specific metrics, such as maximizing sensitivity (catching red flags) over general accuracy.
Technical details
-
Dora
280s
A clinical conversational agent that conducts voice calls for post-op follow-ups and pre-op checks, taking time-consuming tasks off clinicians' plates. It is classified as a regulated medical device.
-
Matrix Framework
650s
The simulation framework used to prove product safety before patient exposure. It recreates clinical conversations using an LLM (PatBot) acting as the simulated patient, grounded in real clinical workflows.
-
PatBot
720s
The LLM used within Matrix to simulate a realistic patient's dialogue. The system validates its realism using Patient and Public Involvement (PPI) studies.
-
BevJudge
830s
A second LLM judge that evaluates simulated dialogues against expected behaviors and hazardous scenarios, providing structured output on triggered hazards. It was validated by expert clinicians to match or exceed human performance.
-
Jeppa (Genetic Pareto)
1020s
A prompt optimization tool used to automatically update prompts iteratively. It optimizes the system's behavior based on a defined metric and cost matrix, making the process reproducible and scalable.
Mentioned resources
- Ufonia
- jossy.co.za
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.