# Simon Obstbaum & Rob Willoughby - Why evals are hard and how we're solving it - AI Native DevCon Jun

## Executive summary

This session introduces advanced methods for evaluating AI agents, arguing that relying solely on 'output evals' (what came out) is insufficient. The focus must shift to 'trajectory evals,' which measure whether the agent followed the correct steps and utilized the right tools. By instrumenting agent behavior—specifically through structured skills and context—teams can significantly improve metrics like PR throughput, decrease cognitive complexity, and ensure adherence to unique organizational conventions (e.g., internal API choices or security policies).

## Key takeaways

- Shift from Output Evals to Trajectory Evals: Evaluating agents requires measuring not just the final output, but also whether the agent activated the correct skills and followed the intended workflow (trajectory) [0:35:46]. Separating activation, trajectory, and outcome is essential for optimizing performance.
- Structured Context Improves Code Quality: The analysis shows that moving from unstructured (L1) to structured context (L3) significantly improves code quality metrics. Specifically, L2 and L3 teams show increased PR throughput, decreased revert rates, and lower cognitive complexity compared to L1 [0:58:46].
- Instruction Following is the Key Differentiator: While task completion may remain high regardless of structure, 'instruction following' (grounded in skills) measures adherence to unique organizational rules. This metric shows the biggest lift and represents the value of encoding proprietary business IP into the agent's context [1:03:52].
- The System, Not Just the Model, Matters: Performance is highly dependent on the entire system stack. Testing must account for the specific model harness (e.g., Opus 4 8 in Claude Code vs. OpenHands), as changing the harness can move scores by up to 100% [1:09:52].

## Technical details

- Agent Evaluation Frameworks: The session distinguishes between Output Evals (final result) and Trajectory Evals (steps taken). The framework emphasizes measuring: 1) Skill Activation, 2) Workflow Sequence (Trajectory), and 3) Final Outcome [0:58:46].
- Productivity Measurement Methodology: The Sweeper Lab study uses a large dataset of ~150,000 engineers and measures productivity by having expert panels rate code for quality, maintainability, and complexity. This process was then replicated using an ML model to scale the analysis [0:23:46].
- Structural Levels (L1-L3): The analysis categorizes repositories based on AI configuration: L1 (no AI config) shows poor change metrics; L2 and L3 show significant improvements in PR throughput, reduced code duplication, and lower cognitive complexity due to structured context/skills [0:58:46].
- Instruction Following vs. Goal Completion: Goal completion (e.g., adding a new API route) can remain high even without structure, but 'instruction following' measures adherence to unique conventions like internal code style guidelines, required back-end services, or prohibited patterns [1:03:52].

## Practical implications

- When building agents, prioritize encoding unique organizational constraints (e.g., internal API standards, security protocols) into skills rather than relying on general model knowledge.
- Implement evaluation pipelines that track skill activation and step-by-step workflow adherence, not just the final code output.
- Recognize that the choice of model harness is as critical to performance metrics as the underlying LLM itself.

## Topics

AI Agents, Software Engineering Productivity, Machine Learning Evaluation, Build Automation, Code Quality Metrics, tessal.io/ainativedev, Stanford AI Spend Index

Source: https://www.youtube.com/watch?v=4d3-Zrmf9Wo