Baz, Docker & Meta on Verifying Agent Code
Summary
The session discusses the critical limitation of current AI coding agents: while they excel at generating code, they often fail at verifying that the code meets complex specifications. Speakers from Baz, Docker, Meta, and Christopher Batey presented advanced strategies—such as using real systems as test oracles, implementing anti-test slop initiatives, and elevating human review to Architectural Decision Records (ADRs)—to ensure reliability and maintain system integrity when integrating AI-generated code.
Key takeaways
-
Verification is the Hard Part
The primary challenge with coding agents is not writing code, but proving that the generated code accurately implements the required specification. Agents are optimized for feature generation, not full spec extraction or verification.
-
Specifying Bugs, Not Features
Baz demonstrated that providing explicit evidence of a bug (e.g., a screen recording of an overlapping button) is far more effective than general specs, as agents may fail to replicate specific, subtle behavioral constraints.
-
Test Oracles Over 100% Coverage
Attempting to achieve 100% test coverage is inefficient. Instead, pointing tests at a real, stable system (a 'test oracle,' like S3) and running a limited set of critical tests can effectively lock down behavior and validate system interactions.
-
Anti Test Slop Initiative
Meta implemented an anti test slop initiative, using a separate AI tool to judge the quality and necessity of AI-generated tests, thereby reducing CI overhead from useless or redundant tests.
-
Elevating Human Review to ADRs
When reviewing large pull requests (e.g., 7,000 lines), human review must move earlier into the process. Architectural Decision Records (ADRs) should capture system-level decisions, allowing agents to check implementations against a structured, human-reviewed architectural baseline.
Technical details
-
Baz Spec Verification
0s
Baz emphasizes that agents are excellent at generating code but struggle to fully extract detailed specifications. They recommend attaching recordings of specific bugs to ensure the generated feature avoids repeating the exact failure.
-
Docker/Rust Testing Strategy
0s
For a 350,000-line Rust codebase, chasing 100% test coverage was ineffective. The most reliable method was using a real system (S3) as a test oracle, running a focused set of 1,500 tests to validate behavior.
-
Meta Anti Test Slop
0s
Meta's initiative uses AI to validate changes on CI by judging the quality of AI-generated tests, preventing the accumulation of redundant or useless test overhead.
-
Architectural Decision Records (ADRs)
0s
ADRs should be the primary artifact for documenting system-level decisions. Humans must dedicate time to reviewing ADRs (ideally with diagrams) before implementation, allowing agents to verify subsequent code against a robust, agreed-upon architectural contract.
Mentioned resources
- Baz
- Docker
- S3
- Meta Workplace
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.