The Hidden 50% Drop in AI Agents Following Your Rules
Summary
The increasing complexity of multi-agent AI coding systems has led to a critical loss of control, evidenced by a reported 50% degradation in agents' adherence to static instruction files like `AGENTS.md` and `CLAUDE.md` [00:14:28]. The talk argues that traditional agile rituals are being replaced by structured, technical controls: detailed specifications (specs), automated verification steps, and advanced merge tactics (like merge queues). To maintain control, developers must move beyond plain text instructions and adopt structured rule sets, such as those used by CodeRabbit, which force adherence across different models.
Key takeaways
-
50% Drop in Agent Adherence
10:28
Baz's data shows a severe, almost overnight, degradation in the usage of static instruction files (`AGENTS.md`, `CLAUDE.md`) by coding agents, suggesting that model releases can break steering capabilities [00:14:28].
-
The Shift from Rituals to Structure
19:12
The bottleneck in software development has shifted from human capacity (PR bottleneck) to system consistency. The process is now being governed by three technical pillars: detailed specs (written in Markdown or linked to issues), automated verification, and advanced merge tactics [00:19:32].
-
Structured Rules Outlast Plain Files
24:20
Structured rule sets (e.g., CodeRabbit's JSON rule set) are significantly more effective at forcing agent adherence than plain instruction files (`CLAUDE.md`) because they provide a stronger, more consistent constraint across models [00:23:40].
-
Long-Horizon Tasks are More Consistent
26:30
While short tasks show high variability, long-horizon code sweeps demonstrate a larger likelihood of agents adhering to correct instructions due to the sheer number of turns and iterations, though users currently prefer faster, shorter loops [00:23:40].
Technical details
-
Agent Steering and Context Management
544s
Coding agents are guided by various 'steering files' (e.g., `AGENTS.md`, `CLAUDE.md`, skills files). Model families (like Opus 3.7) have changed how they ingest this context, sometimes compacting instruction files or inserting skills in separate parts of the context window [00:09:04].
-
Agent Architecture and Data Collection
650s
Baz operates a fleet of domain-specific agents (e.g., security reviewers) that use a 'prompt factory' to continuously define relevant prompts. The system collects data by tracking developer responses (acceptance/rejection) to agent suggestions, labeling success or failure for specific prompts and contexts [00:11:50].
-
Modern Build/DevOps Practices
1152s
The replacement of agile rituals is marked by: 1) **Specs:** Detailed planning files (Markdown/Linear issues). 2) **Verification:** Implementing tools (e.g., GitHub Actions) to suppress bad behaviors resulting from 'steer slips.' 3) **Merge Tactics:** Utilizing native features like Merge Queues for better code inclusion control [00:19:32].
Mentioned resources
- Tessl's skills registry
- baz.ai
- CodeRabbit
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.