# Guardrails First: Engineering Member-Facing Health AI — Rashi Agrawal, Hinge Health

## Executive summary

The talk outlines critical architectural guardrails necessary for deploying member-facing healthcare AI. The core argument is that most safety failures are not model flaws but architectural decisions made before any tokens are generated. Safety must be built into three non-negotiable foundations: protecting PHI at the pipeline boundary, ensuring deterministic code layers handle high-stakes decisions (like emergency routing), and implementing continuous monitoring using multiple signal sources.

## Key takeaways

- Architectural Failures vs. Model Failures: Most AI safety failures in healthcare are architectural decisions, not model failures. The system must be designed to prevent failure at the structural level before considering prompt engineering.
- Three Non-Negotiable Foundations: 1) Constraint is the architecture (not just policy). 2) Deterministic rules must belong above the model layer, as anything that can never be wrong cannot be left to probability. 3) Safety must be a continuous evaluation layer, not a one-time gate.
- PHI Protection at Ingestion: Instead of treating PHI redaction as a runtime problem (on the dashboard), the architecture must strip PHI at the pipeline boundary during ingestion, ensuring it is never stored in the data lake.
- Deterministic Code Layer for High Stakes: Irreversible decisions (e.g., emergency escalation to 911/988, intent routing) must be handled by a deterministic code layer that runs *before* the LLM processes the turn. The model should not get a vote on high-stakes calls.
- Decision Framework: Worst Case Wins: When stakeholders disagree on a feature launch, severity must be set by the worst plausible outcome (the maximum potential harm), not the average case or current capacity. When unsure, default to the safer mistake.

## Technical details

- PHI Handling: To protect PHI, strip it at the pipeline boundary during ingestion rather than relying on reactive redaction when data is viewed in a dashboard. Production and non-production environments must remain completely separate.
- Guardrail Architecture (Code Layer): High-stakes decisions require a deterministic code layer that runs first, forming an 'authority hierarchy' above the LLM. This layer handles irreversible actions like emergency escalation or multi-agentic intent routing.
- Continuous Safety Monitoring: Safety requires continuous evaluation using three signal sources: 1) Automated judges (scoring dimensions like clinical accuracy, safety, drift); 2) Member feedback (the 'truth signal'); and 3) Random sample traces for high-stake cases.
- Decisioning Frameworks: When launching features, use the 'Worst Case Always Wins' rule to set severity. When unsure, apply an 'Asymmetric Default,' meaning always holding/fixing for safety bugs, but shipping for polish bugs.

## Practical implications

- Design systems starting with regulatory compliance (e.g., HIPAA) as the grounding input, allowing the architecture to grow around it.
- Treat all high-stakes decisions as requiring a deterministic code layer that executes before model inference.
- Implement continuous monitoring pipelines capable of ingesting and scoring signals from automated judges, user feedback, and random sample traces.
- Establish clear decision frameworks (like 'Worst Case Wins') for product teams when balancing safety risks against development velocity.

## Topics

AI Architecture, Healthcare Technology, LLM Safety, Data Privacy, Build Engineering, Mount Sinai, ECRI (The patient safety group), HIPAA

Source: https://www.youtube.com/watch?v=YXEqC05WEI0