# Shipping AI to a Million Patients Without an A/B Test — Jared Joselowitz, Ufonia

## Executive summary

The talk details how Ufonia built a comprehensive safety and evaluation stack for Dora, a conversational AI used in clinical post-op follow-ups. Because randomized A/B testing is unethical and illegal when dealing with patients, the system cannot rely on reactive rollbacks or standard model benchmarks. Instead, the approach shifts to rigorous simulation (the 'inner loop') using frameworks like Matrix, which employs simulated patients (PatBot) and an expert LLM judge (BevJudge). Safety is proven by optimizing prompts against a cost matrix (e.g., prioritizing sensitivity over overall accuracy) and utilizing automated prompt optimizers like Jeppa, ensuring the system ships evidence, not just a model.

## Key takeaways

- Safety Constraints in Healthcare AI: Standard software safety nets (A/B testing, rollbacks) fail when dealing with patients because randomizing into a worse variant is unethical and illegal; once a call is made, it cannot be undone. The model card's benchmark scores are insufficient defense at post-incident reviews.
- The Necessity of Simulation: Since real-world testing (the 'outer loop') is too risky, the process must emulate high-reliability industries like self-driving cars. The simulation framework, Matrix, uses an LLM (PatBot) to play the patient against hazards written by clinicians.
- Automated Hazard Detection: A second LLM, BevJudge, validates simulated dialogues. It is trained and validated against a corpus of 240 examples labeled by 10 clinicians from 10 specialties, achieving expert-level performance (e.g., F1 score of 0.96) with near-perfect sensitivity.
- Optimizing Prompts via Cost Matrix: Instead of manual prompt engineering, the process uses optimizers like Jeppa (Genetic Pareto), which iteratively updates prompts based on a defined cost matrix. This allows optimization for specific metrics, such as maximizing sensitivity (catching red flags) over general accuracy.

## Technical details

- Dora: A clinical conversational agent that conducts voice calls for post-op follow-ups and pre-op checks, taking time-consuming tasks off clinicians' plates. It is classified as a regulated medical device.
- Matrix Framework: The simulation framework used to prove product safety before patient exposure. It recreates clinical conversations using an LLM (PatBot) acting as the simulated patient, grounded in real clinical workflows.
- PatBot: The LLM used within Matrix to simulate a realistic patient's dialogue. The system validates its realism using Patient and Public Involvement (PPI) studies.
- BevJudge: A second LLM judge that evaluates simulated dialogues against expected behaviors and hazardous scenarios, providing structured output on triggered hazards. It was validated by expert clinicians to match or exceed human performance.
- Jeppa (Genetic Pareto): A prompt optimization tool used to automatically update prompts iteratively. It optimizes the system's behavior based on a defined metric and cost matrix, making the process reproducible and scalable.

## Practical implications

- For build engineers working on regulated systems, the focus must shift from reactive monitoring (watching dashboards and rolling back) to proactive simulation of failure modes.
- Safety evaluation should be defined by a 'cost matrix' that prioritizes specific harms (e.g., sensitivity for red flags) rather than relying solely on overall accuracy scores.
- The development process is a flywheel: every deployment generates data, which feeds the optimization loop to improve safety and performance.

## Topics

AI Safety, LLM Evaluation, Prompt Engineering, Regulated Medical Devices, Simulation Modeling, Ufonia, jossy.co.za

Source: https://www.youtube.com/watch?v=McknwOzbmyg