Shipping AI to a Million Patients Without an A/B Test — Jared Joselowitz, Ufonia
The talk details how Ufonia built a comprehensive safety and evaluation stack for Dora, a conversational AI used in clinical post-op follow-ups. Because randomized A/B testing is unethical and illegal when dealing with patients, the system cannot rely on reactive rollbacks or standard model benchmarks. Instead, the approach shifts to rigorous simulation (the 'inner loop') using frameworks like Matrix, which employs simulated patients (PatBot) and an expert LLM judge (BevJudge). Safety is proven by optimizing prompts against a cost matrix (e.g., prioritizing sensitivity over overall accuracy) and utilizing automated prompt optimizers like Jeppa, ensuring the system ships evidence, not just a model.
Key takeaways
-
Safety Constraints in Healthcare AI
3:50
Standard software safety nets (A/B testing, rollbacks) fail when dealing with patients because randomizing into a worse variant is unethical and illegal; once a call is made, it cannot be undone. The model card's benchmark scores are insufficient defense at post-incident reviews.
-
The Necessity of Simulation
10:50
Since real-world testing (the 'outer loop') is too risky, the process must emulate high-reliability industries like self-driving cars. The simulation framework, Matrix, uses an LLM (PatBot) to play the patient against hazards written by clinicians.
-
Automated Hazard Detection
13:50
A second LLM, BevJudge, validates simulated dialogues. It is trained and validated against a corpus of 240 examples labeled by 10 clinicians from 10 specialties, achieving expert-level performance (e.g., F1 score of 0.96) with near-perfect sensitivity.
-
Optimizing Prompts via Cost Matrix
17:00
Instead of manual prompt engineering, the process uses optimizers like Jeppa (Genetic Pareto), which iteratively updates prompts based on a defined cost matrix. This allows optimization for specific metrics, such as maximizing sensitivity (catching red flags) over general accuracy.