Guardrails First: Engineering Member-Facing Health AI — Rashi Agrawal, Hinge Health
The talk outlines critical architectural guardrails necessary for deploying member-facing healthcare AI. The core argument is that most safety failures are not model flaws but architectural decisions made before any tokens are generated. Safety must be built into three non-negotiable foundations: protecting PHI at the pipeline boundary, ensuring deterministic code layers handle high-stakes decisions (like emergency routing), and implementing continuous monitoring using multiple signal sources.
Key takeaways
-
Architectural Failures vs. Model Failures
Most AI safety failures in healthcare are architectural decisions, not model failures. The system must be designed to prevent failure at the structural level before considering prompt engineering.
-
Three Non-Negotiable Foundations
3:55
1) Constraint is the architecture (not just policy). 2) Deterministic rules must belong above the model layer, as anything that can never be wrong cannot be left to probability. 3) Safety must be a continuous evaluation layer, not a one-time gate.
-
PHI Protection at Ingestion
8:47
Instead of treating PHI redaction as a runtime problem (on the dashboard), the architecture must strip PHI at the pipeline boundary during ingestion, ensuring it is never stored in the data lake.
-
Deterministic Code Layer for High Stakes
13:35
Irreversible decisions (e.g., emergency escalation to 911/988, intent routing) must be handled by a deterministic code layer that runs *before* the LLM processes the turn. The model should not get a vote on high-stakes calls.
-
Decision Framework: Worst Case Wins
When stakeholders disagree on a feature launch, severity must be set by the worst plausible outcome (the maximum potential harm), not the average case or current capacity. When unsure, default to the safer mistake.