# Three Questions Every Agent Engineer Should Be Able To Answer

## Executive summary

The video outlines a framework for 'owning your own intelligence' when building AI agents, centered on answering three critical questions: where the agent fails, how to reliably reproduce the failure, and what steps are needed to prevent future failures. The methodology emphasizes moving beyond simple input/output testing by creating robust, production-mimicking 'worlds' and 'tasks' using world specifications. Improvements must be implemented across both the agent's harness (e.g., prompts, tool schemas) and the underlying model layer.

## Key takeaways

- The Three Questions of Agent Ownership: To own an agent's intelligence, engineers must master understanding where the agent fails, reliably reproducing that failure, and defining the necessary fixes to prevent recurrence. (0:38)
- Mining Production Traces for Failures: Understanding agent failure requires strong tracing, often achieved by mining production traces. Due to the high cost of frontier LLM-as-judge evaluators at scale, using fine-tuned, evaluation-specific judges to classify perceived errors is a scalable approach. (1:30, 2:27)
- Reproducing Failures with World Specs and Tasks: Simple input/output testing is insufficient because agent behavior is influenced by external factors (e.g., database state, permissions). The solution is to define a 'world spec' that mocks the agent's production environment, and create specific 'tasks' within that world to test emergent behaviors. (4:10, 5:25)
- Layered Fixes for Improvement: Fixing failures requires changes at two levels: the harness (e.g., prompt changes, tool schema updates, context injection) and the model layer (e.g., post-training techniques). (11:08)

## Technical details

- Agent Monitoring and Evaluation: For large-scale monitoring, the speaker suggests using fine-tuned judges to classify agent behavior (e.g., 'delightful experience' vs. 'perceived error') from production traces, offering a scalable alternative to expensive frontier LLM-as-judges. (2:27)
- World Specifications and Tasks: A 'world spec' defines immutable elements of the agent's environment (e.g., permissions, domain-specific rules, file system state). 'Tasks' are specific actions designed to test the agent's behavior within that defined world. (5:25)
- LangChain GTM Agent Example: The LangChain GTM agent uses a world encompassing systems like Gong, Salesforce, and Calendly. A task might be 'research all Gong transcripts for a customer and draft a next best reach-out.' (6:20)
- Data-Driven Improvement: The most effective method for building worlds and tasks is turning production data (telemetry/traces) into structured tasks and worlds, allowing app teams to capture 'spiky edges' of real user interaction that frontier labs cannot. (8:41)

## Practical implications

- Shift testing focus from simple input/output pairs to simulating the full, complex environment (the 'world') the agent operates in.
- Implement robust tracing and monitoring using specialized, fine-tuned judges to manage the cost and scale of agent evaluation.
- Structure agent development by defining explicit 'world specs' and 'tasks' to ensure that improvements are quantifiable and target specific failure modes observed in production data.
- Recognize that agent improvement requires coordinated changes across the application harness (prompts, tools) and the underlying model layer.

## Topics

AI Agents, LLM Evaluation, Production Tracing, World Modeling, Build Engineering, LangChain, Continual Learning, How we built LangChain's GTM Agent, LangSmith Observability

Source: https://www.youtube.com/watch?v=oVKGKdIAgtg