# Baz, Docker & Meta on Verifying Agent Code

## Executive summary

The session discusses the critical limitation of current AI coding agents: while they excel at generating code, they often fail at verifying that the code meets complex specifications. Speakers from Baz, Docker, Meta, and Christopher Batey presented advanced strategies—such as using real systems as test oracles, implementing anti-test slop initiatives, and elevating human review to Architectural Decision Records (ADRs)—to ensure reliability and maintain system integrity when integrating AI-generated code.

## Key takeaways

- Verification is the Hard Part: The primary challenge with coding agents is not writing code, but proving that the generated code accurately implements the required specification. Agents are optimized for feature generation, not full spec extraction or verification.
- Specifying Bugs, Not Features: Baz demonstrated that providing explicit evidence of a bug (e.g., a screen recording of an overlapping button) is far more effective than general specs, as agents may fail to replicate specific, subtle behavioral constraints.
- Test Oracles Over 100% Coverage: Attempting to achieve 100% test coverage is inefficient. Instead, pointing tests at a real, stable system (a 'test oracle,' like S3) and running a limited set of critical tests can effectively lock down behavior and validate system interactions.
- Anti Test Slop Initiative: Meta implemented an anti test slop initiative, using a separate AI tool to judge the quality and necessity of AI-generated tests, thereby reducing CI overhead from useless or redundant tests.
- Elevating Human Review to ADRs: When reviewing large pull requests (e.g., 7,000 lines), human review must move earlier into the process. Architectural Decision Records (ADRs) should capture system-level decisions, allowing agents to check implementations against a structured, human-reviewed architectural baseline.

## Technical details

- Baz Spec Verification: Baz emphasizes that agents are excellent at generating code but struggle to fully extract detailed specifications. They recommend attaching recordings of specific bugs to ensure the generated feature avoids repeating the exact failure.
- Docker/Rust Testing Strategy: For a 350,000-line Rust codebase, chasing 100% test coverage was ineffective. The most reliable method was using a real system (S3) as a test oracle, running a focused set of 1,500 tests to validate behavior.
- Meta Anti Test Slop: Meta's initiative uses AI to validate changes on CI by judging the quality of AI-generated tests, preventing the accumulation of redundant or useless test overhead.
- Architectural Decision Records (ADRs): ADRs should be the primary artifact for documenting system-level decisions. Humans must dedicate time to reviewing ADRs (ideally with diagrams) before implementation, allowing agents to verify subsequent code against a robust, agreed-upon architectural contract.

## Practical implications

- Shift human effort from reviewing massive pull requests to rigorously defining and reviewing Architectural Decision Records (ADRs).
- When testing AI-generated code, prioritize using real, stable systems (test oracles) over achieving theoretical 100% test coverage.
- Adopt tooling (like Meta's anti test slop) that uses AI to vet the quality and necessity of generated tests, reducing CI overhead.
- For complex specifications, supplement written specs with concrete examples, such as video recordings of specific bugs to avoid.

## Topics

AI Development, Software Testing, Build Engineering, CI/CD, Architecture Design, Code Review, Baz, Docker, S3, Meta Workplace

Source: https://www.youtube.com/watch?v=DFDyRt4cU1Q