# The Hidden 50% Drop in AI Agents Following Your Rules

## Executive summary

The increasing complexity of multi-agent AI coding systems has led to a critical loss of control, evidenced by a reported 50% degradation in agents' adherence to static instruction files like `AGENTS.md` and `CLAUDE.md` [00:14:28]. The talk argues that traditional agile rituals are being replaced by structured, technical controls: detailed specifications (specs), automated verification steps, and advanced merge tactics (like merge queues). To maintain control, developers must move beyond plain text instructions and adopt structured rule sets, such as those used by CodeRabbit, which force adherence across different models.

## Key takeaways

- 50% Drop in Agent Adherence: Baz's data shows a severe, almost overnight, degradation in the usage of static instruction files (`AGENTS.md`, `CLAUDE.md`) by coding agents, suggesting that model releases can break steering capabilities [00:14:28].
- The Shift from Rituals to Structure: The bottleneck in software development has shifted from human capacity (PR bottleneck) to system consistency. The process is now being governed by three technical pillars: detailed specs (written in Markdown or linked to issues), automated verification, and advanced merge tactics [00:19:32].
- Structured Rules Outlast Plain Files: Structured rule sets (e.g., CodeRabbit's JSON rule set) are significantly more effective at forcing agent adherence than plain instruction files (`CLAUDE.md`) because they provide a stronger, more consistent constraint across models [00:23:40].
- Long-Horizon Tasks are More Consistent: While short tasks show high variability, long-horizon code sweeps demonstrate a larger likelihood of agents adhering to correct instructions due to the sheer number of turns and iterations, though users currently prefer faster, shorter loops [00:23:40].

## Technical details

- Agent Steering and Context Management: Coding agents are guided by various 'steering files' (e.g., `AGENTS.md`, `CLAUDE.md`, skills files). Model families (like Opus 3.7) have changed how they ingest this context, sometimes compacting instruction files or inserting skills in separate parts of the context window [00:09:04].
- Agent Architecture and Data Collection: Baz operates a fleet of domain-specific agents (e.g., security reviewers) that use a 'prompt factory' to continuously define relevant prompts. The system collects data by tracking developer responses (acceptance/rejection) to agent suggestions, labeling success or failure for specific prompts and contexts [00:11:50].
- Modern Build/DevOps Practices: The replacement of agile rituals is marked by: 1) **Specs:** Detailed planning files (Markdown/Linear issues). 2) **Verification:** Implementing tools (e.g., GitHub Actions) to suppress bad behaviors resulting from 'steer slips.' 3) **Merge Tactics:** Utilizing native features like Merge Queues for better code inclusion control [00:19:32].

## Practical implications

- When building an AI-native workflow, do not rely solely on plain instruction files (`.md`). Implement structured rule sets (like JSON) to ensure consistent agent behavior across model updates.
- Rethink collaboration by focusing on explicit, machine-readable specifications and automated verification gates rather than relying on human-led agile rituals.
- Use tools like the Rework Calculator (available via baz.ai) to quantify the financial cost of inconsistent agent activity and redundant workflows.

## Topics

AI Agents, Software Development Lifecycle (SDLC), Build Engineering, DevOps, Code Review, Context Management, Tessl's skills registry, baz.ai, CodeRabbit

Source: https://www.youtube.com/watch?v=R1j6HtykHNE