Baz, Docker & Meta on Verifying Agent Code
The session discusses the critical limitation of current AI coding agents: while they excel at generating code, they often fail at verifying that the code meets complex specifications. Speakers from Baz, Docker, Meta, and Christopher Batey presented advanced strategies—such as using real systems as test oracles, implementing anti-test slop initiatives, and elevating human review to Architectural Decision Records (ADRs)—to ensure reliability and maintain system integrity when integrating AI-generated code.
Key takeaways
-
Verification is the Hard Part
The primary challenge with coding agents is not writing code, but proving that the generated code accurately implements the required specification. Agents are optimized for feature generation, not full spec extraction or verification.
-
Specifying Bugs, Not Features
Baz demonstrated that providing explicit evidence of a bug (e.g., a screen recording of an overlapping button) is far more effective than general specs, as agents may fail to replicate specific, subtle behavioral constraints.
-
Test Oracles Over 100% Coverage
Attempting to achieve 100% test coverage is inefficient. Instead, pointing tests at a real, stable system (a 'test oracle,' like S3) and running a limited set of critical tests can effectively lock down behavior and validate system interactions.
-
Anti Test Slop Initiative
Meta implemented an anti test slop initiative, using a separate AI tool to judge the quality and necessity of AI-generated tests, thereby reducing CI overhead from useless or redundant tests.
-
Elevating Human Review to ADRs
When reviewing large pull requests (e.g., 7,000 lines), human review must move earlier into the process. Architectural Decision Records (ADRs) should capture system-level decisions, allowing agents to check implementations against a structured, human-reviewed architectural baseline.