Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Summary
The session addresses the critical challenge of trusting autonomous coding agents. Rather than relying solely on code review (PRs), the focus shifts to building robust verification loops that allow agents to prove their work is complete and correct. Key strategies include implementing CLI health checks, using browser snapshots, and integrating LLM judges to create self-healing, deterministic development pipelines. This approach transforms agent failures into actionable data that drives the next fix, significantly accelerating development velocity and scaling agent usage.
Key takeaways
-
Verification is Foundational to Trust
As model capabilities advance, verification (reliability and evals) becomes the most critical foundational element. The goal is to move beyond simply checking the code and instead prove the desired *result* or *state* has been achieved.
-
The Need for Self-Correction
47:21
Agents must be able to determine when they have failed and initiate the necessary steps to fix themselves, rather than requiring constant human intervention. This self-healing capability is key to scaling agent-driven development.
-
Prioritizing Deterministic Proofs
1:00:29
The most reliable form of proof comes from deterministic checks (e.g., CLI commands, unit tests, API calls) rather than relying on vague prompts or non-deterministic LLM outputs. Tools like Playwright and dedicated CLI verifiers are essential for this.
Technical details
-
Verification Loops and Tools
1731s
The workshop demonstrates building verification loops using a tool called `snack`. This tool allows agents to perform multiple checks: `pnpm snack doctor` (CLI health checks), `pnpm snack snapshot` (capturing page state in JSON), `pnpm snack join` (simulating user flow), and `pnpm snack prove` (running deterministic checks).
-
System Architecture for Agents
2200s
The concept of a centralized authority for verification is explored using a service like CTX (Context/Cross-Agent Harness Memory Layer). This service acts as a single point of truth, receiving agent reports and running hardened verifiers (e.g., checking the GitHub API for PR status) before marking a workflow node as satisfied.
-
LLM Evaluation and State Management
3400s
To handle complex state changes (e.g., an email being saved to a waitlist), the system must go beyond simple UI checks. Techniques include checking local storage, network logs, and using LLM judges to compare observed states against expected outcomes, even when the confirmation text changes (the 'contract' breaks).
-
Development Methodology
2500s
The recommended approach is decomposing large tasks into small, clean, and verifiable steps. This allows agents to complete one task, verify its state, and then pass that verified state to the next agent/task, minimizing risk and maximizing parallelization.
Mentioned resources
- WorkOS
- WorkOS Workshop Repo
- Midnight Snack Club Demo App
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.