The Video Signal

AI Native Dev

Simon Obstbaum & Rob Willoughby - Why evals are hard and how we're solving it - AI Native DevCon Jun

Published 2026-07-27 · Duration 36:39

Summary

This session introduces advanced methods for evaluating AI agents, arguing that relying solely on 'output evals' (what came out) is insufficient. The focus must shift to 'trajectory evals,' which measure whether the agent followed the correct steps and utilized the right tools. By instrumenting agent behavior—specifically through structured skills and context—teams can significantly improve metrics like PR throughput, decrease cognitive complexity, and ensure adherence to unique organizational conventions (e.g., internal API choices or security policies).

Download summary

Key takeaways

  1. Shift from Output Evals to Trajectory Evals 29:56

    Evaluating agents requires measuring not just the final output, but also whether the agent activated the correct skills and followed the intended workflow (trajectory) [0:35:46]. Separating activation, trajectory, and outcome is essential for optimizing performance.

  2. Structured Context Improves Code Quality 5:46

    The analysis shows that moving from unstructured (L1) to structured context (L3) significantly improves code quality metrics. Specifically, L2 and L3 teams show increased PR throughput, decreased revert rates, and lower cognitive complexity compared to L1 [0:58:46].

  3. Instruction Following is the Key Differentiator 6:32

    While task completion may remain high regardless of structure, 'instruction following' (grounded in skills) measures adherence to unique organizational rules. This metric shows the biggest lift and represents the value of encoding proprietary business IP into the agent's context [1:03:52].

  4. The System, Not Just the Model, Matters 10:52

    Performance is highly dependent on the entire system stack. Testing must account for the specific model harness (e.g., Opus 4 8 in Claude Code vs. OpenHands), as changing the harness can move scores by up to 100% [1:09:52].

Technical details

  • Agent Evaluation Frameworks 1796s

    The session distinguishes between Output Evals (final result) and Trajectory Evals (steps taken). The framework emphasizes measuring: 1) Skill Activation, 2) Workflow Sequence (Trajectory), and 3) Final Outcome [0:58:46].

  • Productivity Measurement Methodology 84s

    The Sweeper Lab study uses a large dataset of ~150,000 engineers and measures productivity by having expert panels rate code for quality, maintainability, and complexity. This process was then replicated using an ML model to scale the analysis [0:23:46].

  • Structural Levels (L1-L3) 346s

    The analysis categorizes repositories based on AI configuration: L1 (no AI config) shows poor change metrics; L2 and L3 show significant improvements in PR throughput, reduced code duplication, and lower cognitive complexity due to structured context/skills [0:58:46].

  • Instruction Following vs. Goal Completion 392s

    Goal completion (e.g., adding a new API route) can remain high even without structure, but 'instruction following' measures adherence to unique conventions like internal code style guidelines, required back-end services, or prohibited patterns [1:03:52].

Mentioned resources

  • tessal.io/ainativedev (Website/Tool)
  • Stanford AI Spend Index (Index/Data Set)

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.