Guardrails First: Engineering Member-Facing Health AI — Rashi Agrawal, Hinge Health
Summary
The talk outlines critical architectural guardrails necessary for deploying member-facing healthcare AI. The core argument is that most safety failures are not model flaws but architectural decisions made before any tokens are generated. Safety must be built into three non-negotiable foundations: protecting PHI at the pipeline boundary, ensuring deterministic code layers handle high-stakes decisions (like emergency routing), and implementing continuous monitoring using multiple signal sources.
Key takeaways
-
Architectural Failures vs. Model Failures
Most AI safety failures in healthcare are architectural decisions, not model failures. The system must be designed to prevent failure at the structural level before considering prompt engineering.
-
Three Non-Negotiable Foundations
3:55
1) Constraint is the architecture (not just policy). 2) Deterministic rules must belong above the model layer, as anything that can never be wrong cannot be left to probability. 3) Safety must be a continuous evaluation layer, not a one-time gate.
-
PHI Protection at Ingestion
8:47
Instead of treating PHI redaction as a runtime problem (on the dashboard), the architecture must strip PHI at the pipeline boundary during ingestion, ensuring it is never stored in the data lake.
-
Deterministic Code Layer for High Stakes
13:35
Irreversible decisions (e.g., emergency escalation to 911/988, intent routing) must be handled by a deterministic code layer that runs *before* the LLM processes the turn. The model should not get a vote on high-stakes calls.
-
Decision Framework: Worst Case Wins
When stakeholders disagree on a feature launch, severity must be set by the worst plausible outcome (the maximum potential harm), not the average case or current capacity. When unsure, default to the safer mistake.
Technical details
-
PHI Handling
527s
To protect PHI, strip it at the pipeline boundary during ingestion rather than relying on reactive redaction when data is viewed in a dashboard. Production and non-production environments must remain completely separate.
-
Guardrail Architecture (Code Layer)
815s
High-stakes decisions require a deterministic code layer that runs first, forming an 'authority hierarchy' above the LLM. This layer handles irreversible actions like emergency escalation or multi-agentic intent routing.
-
Continuous Safety Monitoring
1037s
Safety requires continuous evaluation using three signal sources: 1) Automated judges (scoring dimensions like clinical accuracy, safety, drift); 2) Member feedback (the 'truth signal'); and 3) Random sample traces for high-stake cases.
-
Decisioning Frameworks
When launching features, use the 'Worst Case Always Wins' rule to set severity. When unsure, apply an 'Asymmetric Default,' meaning always holding/fixing for safety bugs, but shipping for polish bugs.
Mentioned resources
- Mount Sinai
- ECRI (The patient safety group)
- HIPAA
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.