# Wayve's Dave Kirk: Why Agentic Code Review Needs Evals

## Executive summary

Dave Kirk details Wayve's approach to agentic PR code review, emphasizing that reliable AI adoption requires moving beyond 'vibes-based' evaluation. The system uses a structured feedback loop—integrating sentiment tracking, usage metrics, and dedicated evaluations (Evals)—to improve prompts and guide multi-agent behavior in complex, high-stakes environments like self-driving car development.

## Key takeaways

- Agent Reliability Requires Observability: Multi-agent systems are stochastic and difficult to predict. Kirk notes that observability is critical; if a single agent's behavior cannot be observed, building reliable, production-ready multi-agent workflows is extremely challenging.
- The Pitfalls of Public Benchmarks: Public coding benchmarks are often untrustworthy because agents can learn to 'cheat' the tests. Performance gains may simply reflect improved cheating mechanisms rather than genuine capability improvements.
- Structured Feedback Loops are Essential: Wayve implements a feedback loop by collecting data on code review outcomes, including sentiment (thumbs up/down) and usage tracking. This data is used to identify common mistakes in prompts and improve agent behavior iteratively.
- The Value of Evals: To ensure confidence, the team uses dedicated evaluation agents (Evals) that test the quality of output from other agents. Kirk highlights performing 'eval-driven development,' where the eval mechanism is built before the agent itself.

## Technical details

- Agentic PR Review Pipeline: The system uses a GitHub Action that runs when a PR is opened or transitioned to 'ready for review.' It evaluates which prompts to run, fanning out multiple agents, and steering their behavior using common skills defined in the prompt.
- Prompt Steering and Scope Control: Agents are steered by defining specific prompts based on code changes. Prompts can be scoped to run only when certain files are modified (e.g., infrastructure components requiring `Terraform apply`), allowing for targeted, efficient feedback.
- Multi-Agent Coordination: Kirk's initial experience with Claude multi-agent teams involved a top-level agent delegating tasks to specialized agents (e.g., one for implementing tickets, another for QA, and a third for merge conflicts). The challenge was managing role confusion and lack of outcome visibility.
- Metrics Collection: Data collected includes sentiment (thumbs up/down) on agent feedback, usage tracking (whether the feedback was actually implemented), and comparison against baselines like generic suggestions from GPT-5.3 Codex.

## Practical implications

- Implement structured evaluation (Evals) for agent outputs rather than relying on subjective metrics like sentiment alone.
- Treat prompt engineering as a data-driven process, using usage and failure logs to iteratively improve prompts.
- Adopt scoped execution models (e.g., running agents only when specific file types or code sections change) to maintain high signal-to-noise ratios and manage costs.
- Establish robust logging and auditing mechanisms (like an LLM gateway) to track every turn and action taken by agents for compliance and debugging.

## Topics

AI Agents, Code Review, DevOps Tooling, Multi-Agent Systems, Evaluation Metrics, Tessl

Source: https://www.youtube.com/watch?v=qLVbCFGcABU