Designing Agents (The Floor Is the Frontier) — Ben Hylak, Raindrop
The talk argues that traditional evaluation methods designed for simple chatbots are inadequate for modern AI agents. The focus must shift from merely finding issues to 'raising the floor'—mitigating catastrophic worst-case failures that break user trust (e.g., recommending a competitor or deleting data). Practically, this requires treating agent evaluations as robust code tests rather than relying on prompt playgrounds, and focusing analysis on issue start dates and affected user percentages.
Key takeaways
-
Focus on Raising the Floor (Safety)
17:33
The most critical concern for agents is not their maximum capability (the ceiling), but their worst-case behavior (the floor). The floor represents failures that break user trust, such as accidental data deletion or sending inappropriate communications.
-
Evals Must Be Code, Not Prompts
12:00
Evaluation suites should be implemented as code (like unit or end-to-end tests) rather than relying on managed prompt playgrounds. This approach is necessary because the underlying structure of agents and their tools has fundamentally changed.
-
Prioritize Issue Context
14:00
When tracking issues, knowing only that an issue exists is insufficient. It is critical to track two metrics: when the issue started (to identify recent changes) and what percentage of users are affected.
-
Tactical Analysis Tips
17:05
1. **Clusters are not issues:** Clustering traces is useful for one-off analysis but does not scale well or reliably track changes over time. 2. **Code Mode Scaling:** Apply code mode concepts to traces by writing classifiers and running them in a sandbox at production volume. 3. **Anomaly Investigation:** Agents are poor at detecting anomalies, so instead, surface deterministic signals (like keyword spikes) for the agent to investigate.