Evals-Driven Development for a Mental Health AI Coach — Akele Reed & Dave Revere, SonderMind
SonderMind details its approach to building Sonder, a clinically grounded AI coach for mental health support. The core focus is on achieving safety and effectiveness through eval-driven development, moving beyond general-purpose LLMs which are unsuitable for high-stakes care. The architecture employs modular guardrails (Input/Output) within a Supervisor/Executor/Evaluator framework. Safety is maintained by integrating clinical judgment into the CI pipeline via annotated traces, ensuring that every model or prompt change is scored against expert-defined 'evals' before deployment.
Key takeaways
-
Specialized AI for Mental Health
2:00
General purpose LLMs are insufficient for mental health care due to the high stakes involved. Sonder was designed specifically to act as a front door to human therapy or provide support between sessions, addressing a critical gap in current AI offerings.
-
Robust Architecture and Guardrails
4:00
The system uses separate Input and Output guardrails—acting as 'sandwich' layers—to monitor user input and the entire conversation flow for clinical safety risks. This modularity allows iteration without compromising safety.
-
Clinical Feedback Loop in CI
9:40
Safety is enforced by capturing edge-case conversations (traces) and having licensed professionals annotate them. These annotations generate typed 'evals' that gate releases, ensuring every model or prompt change is tested against real-world clinical judgment.
-
Focus on Correct Triggers
6:10
The guardrails are calibrated not to generate more alerts (false positives), but to correctly identify genuine safety risks. The system must distinguish between a user in an active crisis and one seeking past support.