Inside 847 Production Clinical AI Notes — Sebastian Fox, Composo
This talk addresses the critical challenge of evaluating high-stakes AI systems, particularly ambient scribes in healthcare, where dangerous failures often manifest as subtle omissions or hallucinations rather than obvious errors. The speaker argues that traditional verification methods (like fixed rubrics or simple difference checks) fail because the 'standard of good' is tacit, contextual, and constantly evolving. A proposed solution involves building a continuous evaluation loop: Discovering failure modes from real-world outputs, capturing expert judgment on these modes, and calibrating every new output against this accumulated, case-specific context rather than a static rule set.
Key takeaways
-
High-Stakes Failure Modes
2:09
In clinical notes, the most dangerous failures are often subtle omissions (e.g., missing jaw pain symptoms) or hallucinations that look technically correct but are factually wrong. In large studies, nearly 1 in 20 notes carried an error serious enough to cause significant harm [1:29].
-
Limitations of Current AI Evaluation
11:43
Verification is only easy for the 'easy half' (e.g., spotting differences between transcript and note). The hard part is determining which difference—an omission, change, or addition—actually matters in context [7:03]. This judgment is tacit, contextual, and moving.
-
The Continuous Evaluation Loop
To overcome the limitations of static rubrics, the recommended approach is a continuous loop: 1) Discover failure modes from real outputs (building a 'failure mode ontology'), 2) Capture expert judgment on these modes, and 3) Calibrate every output against this accumulated, case-specific context, rather than relying on fixed weights or prompts [13:49].