Why 80% Reliability Isn't Good Enough — Felipe Blanes, Amazon AGI Lab
Felipe Blanes discusses the 'benchmark illusion' in building AI agents, where high performance on static benchmarks does not guarantee reliability in real-world customer use. He proposes an 'eval flywheel'—a continuous, customer-driven process—that mandates defining success based on customer needs, capturing real-world signals, diagnosing gaps (model, engineering, or product), and feeding those insights back into development decisions. The talk emphasizes that transparency about product limitations is crucial for building customer trust, noting that reliability is a binary 'trust' or 'no trust' state, rather than a linear scale.
Key takeaways
-
The Benchmark Illusion
3:42
Relying solely on static benchmarks or synthetic data (static evals) is insufficient because real customer use cases often involve unexpected actions that cause product failure. The solution is to close the customer loop.
-
The Eval Flywheel
7:21
The core process involves four steps: 1) Define success (based on customer needs, not internal assumptions); 2) Capture signals (using instrumentation and, critically, talking to customers); 3) Diagnose gaps (categorizing issues into model failures, engineering gaps, or product positioning issues); and 4) Feed decisions (prioritizing fixes for the science, engineering, and product teams).
-
The Trust Cliff
15:35
Reliability is not linear. Customers perceive 80% reliability as requiring too much manual monitoring and effort, whereas reaching a threshold (around 90-92%) shifts the perception to 'trustworthy' and allows for full workflow handover.
-
Transparency Builds Trust
It is more effective to be transparent about a product's limitations and what is out of scope than to focus only on high benchmarks. Openness about capabilities and limitations increases customer trust.