I Believe Everything My Agents Tell Me: Three Levels of Review — Corey Gallon, Rexmore
Summary
The video outlines a three-level agent-based review method designed to mitigate common AI agent failure modes: overstating work, misreading facts, and drawing incorrect conclusions. The method establishes a concrete checking workflow, moving beyond simple agent output to verifiable, multi-stage validation. The levels range from artifact verification (Level One) to independent source checking (Level Two), culminating in adversarial, multi-persona review (Level Three). The speaker demonstrates the method's efficacy in complex scenarios, including memory leak debugging, financial accounting reconciliation, and legal demand letter drafting.
Key takeaways
-
Agent Failure Modes
Agents commonly fail by overstating completed work, misreading source facts, or drawing incorrect conclusions, even when using billions of tokens (00:00).
-
Level One: Artifact Verification
2:02
Prevents agents from overstating work. The review requires the agent to provide an artifact verified with a fingerprint, or explicitly state that the artifact is missing, preventing a simple 'done' status (1:22).
-
Level Two: Independent Source Checks
3:23
Addresses factual errors. Two independent agents process the same instructions, noting the source for every fact. A third agent then compares these results line-by-line against the original source to settle all differences (2:03).
-
Level Three: Adversarial and Fresh Review
4:04
Handles reasoning errors. This involves three stages: 1) Self-audit (marking claims as measured, inferred, or assumed); 2) Cold reviewers (personas with reasons to find fault, seeing no history); and 3) Fresh reviewers (new agents checking the final version) (2:44).
Technical details
-
Level One Review
122s
Requires the agent to end its review by providing an artifact verified with a fingerprint, or an explicit 'artifact missing' report with reasoning. The dispatching agent must run the final check itself (1:22).
-
Level Two Review
203s
Uses a 'control total' (e.g., a balance printed on statements) as the source of truth. Two agents work independently, and a third agent compares their outputs against the source to reconcile every difference (2:03).
-
Level Three Review
244s
A multi-stage process for high-stakes reasoning. It combines self-auditing, cold-persona review (reviewers with no original context), and final review by brand new agents (2:44).
-
Case Study: Accounting Reconciliation
657s
Level Two was used to process a year of scanned accounting statements. Two agents extracted data independently, and the process caught 56 differences in descriptions, dates, and notes, resulting in records that balanced to the penny (6:57).
-
Case Study: Demand Letter Drafting
Level Three was applied to a legal demand letter. Cold reviewers identified a three-year auto-renewal clause buried in the contract's fine print, which the original agent's own check had missed (8:53).
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.