The Video Signal

Topic

Code Review

All digests tagged Code Review

Wayve's Dave Kirk: Why Agentic Code Review Needs Evals thumbnail

· 23:55

Wayve's Dave Kirk: Why Agentic Code Review Needs Evals

Dave Kirk details Wayve's approach to agentic PR code review, emphasizing that reliable AI adoption requires moving beyond 'vibes-based' evaluation. The system uses a structured feedback loop—integrating sentiment tracking, usage metrics, and dedicated evaluations (Evals)—to improve prompts and guide multi-agent behavior in complex, high-stakes environments like self-driving car development.

Key takeaways

  1. Agent Reliability Requires Observability 2:08

    Multi-agent systems are stochastic and difficult to predict. Kirk notes that observability is critical; if a single agent's behavior cannot be observed, building reliable, production-ready multi-agent workflows is extremely challenging.

  2. The Pitfalls of Public Benchmarks 10:53

    Public coding benchmarks are often untrustworthy because agents can learn to 'cheat' the tests. Performance gains may simply reflect improved cheating mechanisms rather than genuine capability improvements.

  3. Structured Feedback Loops are Essential 22:30

    Wayve implements a feedback loop by collecting data on code review outcomes, including sentiment (thumbs up/down) and usage tracking. This data is used to identify common mistakes in prompts and improve agent behavior iteratively.

  4. The Value of Evals 23:25

    To ensure confidence, the team uses dedicated evaluation agents (Evals) that test the quality of output from other agents. Kirk highlights performing 'eval-driven development,' where the eval mechanism is built before the agent itself.

Watch on YouTube Full article

Harness Engineering: Building an AI Software Factory thumbnail

· 53:34

Harness Engineering: Building an AI Software Factory

Harness engineering is a discipline focused on building automated loops of checks and maintenance agents that allow teams to delegate increasing amounts of codebase development to AI. The goal is not merely higher velocity, but achieving higher quality by shifting from manual code review (the primary bottleneck) to systematic process oversight. This involves tracking three key dimensions—autonomy, automation, and quality—and implementing layered validation systems: the Inner Loop (unit tests/linters), Outer Loop (agentic QA/UI testing), and Meta Loop (maintenance agents that analyze historical data for systemic improvements).

Key takeaways

  1. The Three Dimensions of Agent Adoption 10:39

    When adopting AI agents, track three metrics: Autonomy (how many human course corrections are needed); Automation (the level of oversight required, indicating trust); and Quality (ensuring the shipped product remains high quality). Progressing requires improving these dimensions sequentially.

  2. The Three Loops of Harness Engineering 19:04

    1. Inner Loop (Autonomy): Focuses on cheap, frequent checks like pedantic linting or unit tests to ensure agents get it right the first time. 2. Outer Loop (Automation): Involves slower, in-depth checks, such as agentic code review or running the product through a UI/CLI. 3. Meta Loop: Utilizes maintenance agents that analyze historical data (CI logs, PR comments) to propose systemic fixes and improvements to the entire process.

  3. The Primary Barrier is Organizational 28:20

    Harness engineering is fundamentally an organizational transformation, not just a technical one. Success requires changing workflows—for example, moving from monolithic PRs to smaller, low-risk chunks that can auto-merge, thereby shifting human behavior toward better practices.

Watch on YouTube Full article

Jack Wotherspoon - Humans vs. Slop: Rewriting the Rules of Open-Source - AI Native DevCon thumbnail

· 32:17

Jack Wotherspoon - Humans vs. Slop: Rewriting the Rules of Open-Source - AI Native DevCon

The rise of powerful AI agents is fundamentally changing open-source development by making code generation nearly free and abundant, leading to a flood of low-quality contributions ('AI slop'). This shift threatens the traditional human-to-human social contract of open source. Maintainers must implement new governance models—such as requiring issues before pull requests (PRs), rate limiting external contributors, and utilizing automation—to manage the influx of code while preserving quality and accountability.

Key takeaways

  1. The Open Source Shift 2:00

    Open-source development is moving from a human-to-human experience to one involving AI agents. The core challenge is that while generating code is cheap (sometimes free), reviewing, maintaining, and trusting the generated code remains expensive and difficult.

  2. Guardrails Against Slop 6:25

    To combat 'drive-by PRs' (where users submit fixes without prior discussion) and uncontrolled agent activity, maintainers should require contributors to file an Issue before submitting a Pull Request. This is cited as eliminating approximately 90% of drive-by PRs.

  3. Rate Limiting Contributions 7:00

    Implement rate limits on external contributors (e.g., limiting the number of active PRs) to prevent automated swarms of nonsensical code submissions, which can overwhelm maintainers.

  4. Trust and Governance Systems 9:40

    New systems are emerging to restore trust: 'Vouch' acts as a referral system for contributors, while 'Open Source Vacation' allows projects or solo maintainers to temporarily halt contributions when needed. Projects should also use context files (like `agents.md`) to guide all AI tools.

  5. Codifying Best Practices with Skills 11:20

    Implementing 'agent skills' (e.g., PR creator, docs writer) into the repository helps enforce best practices—such as running tests and following templates—for both human and AI contributors, thereby raising the overall quality bar.

Watch on YouTube Full article