Inside 847 Production Clinical AI Notes — Sebastian Fox, Composo
Summary
This talk addresses the critical challenge of evaluating high-stakes AI systems, particularly ambient scribes in healthcare, where dangerous failures often manifest as subtle omissions or hallucinations rather than obvious errors. The speaker argues that traditional verification methods (like fixed rubrics or simple difference checks) fail because the 'standard of good' is tacit, contextual, and constantly evolving. A proposed solution involves building a continuous evaluation loop: Discovering failure modes from real-world outputs, capturing expert judgment on these modes, and calibrating every new output against this accumulated, case-specific context rather than a static rule set.
Key takeaways
-
High-Stakes Failure Modes
2:09
In clinical notes, the most dangerous failures are often subtle omissions (e.g., missing jaw pain symptoms) or hallucinations that look technically correct but are factually wrong. In large studies, nearly 1 in 20 notes carried an error serious enough to cause significant harm [1:29].
-
Limitations of Current AI Evaluation
11:43
Verification is only easy for the 'easy half' (e.g., spotting differences between transcript and note). The hard part is determining which difference—an omission, change, or addition—actually matters in context [7:03]. This judgment is tacit, contextual, and moving.
-
The Continuous Evaluation Loop
To overcome the limitations of static rubrics, the recommended approach is a continuous loop: 1) Discover failure modes from real outputs (building a 'failure mode ontology'), 2) Capture expert judgment on these modes, and 3) Calibrate every output against this accumulated, case-specific context, rather than relying on fixed weights or prompts [13:49].
Technical details
-
AI Evaluation Architecture
756s
The speaker advocates for placing a 'checker' model after the generator (LLM) to perform verification. This checker should utilize detailed rubrics with worked examples and potentially deterministic NLP concept counting, but must be dynamic.
-
Contextual Calibration
Instead of a single pre-specified rubric, the system should assemble a 'case-specific standard' for each output. This involves retrieving and providing context from similar past judgments (e.g., nearest cases with the same symptom shape) to guide the judge [13:49].
-
Failure Mode Ontology
The process must start by putting the system in production and clustering observed errors (failure modes) from real outputs to build a comprehensive map of potential failure points, rather than guessing them [13:49].
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.