# Agents Write 95% of Our Code. Here's the Catch

## Executive summary

As AI agents assume control over an estimated 95% of code production in advanced software factories, traditional code review processes are insufficient. The talk introduces the role of the 'harness engineer,' a new skill set focused on system-level controls: defining invariants, performing deep analytics on agent logs and PR data, and implementing fine-grained risk/operations policies (like auto-merge ladders). This shift requires engineers to move from writing code features to building robust guardrails that ensure consistency and quality across agent-driven pipelines.

## Key takeaways

- The Paradox of AI Adoption: While AI coding tool adoption is high, benchmarks are becoming saturated. Concurrently, the number of reported bugs and incidents is rising, indicating that agents may generate code that lacks maintainability or systemic health (00:15:24).
- The Rise of the Harness Engineer: Engineering focus must shift from pure feature building to defining and enforcing system invariants. The three critical new skill sets are Systems Thinking, Analytics, and Risk/Operations (00:09:34).
- Instruction Following Gap in Skills: Tessl's internal skills benchmark revealed that while agents achieved high task completion rates, they only followed approximately 70% of the total instructions defined within a skill (00:08:22).
- Systemic Control through Invariants and CI Gates: Engineers must identify general principles (invariants)—such as design system rules or desired code structure—and encode them into deterministic checks, verifiers, or CI gates to ensure consistency across the codebase (00:12:56).

## Technical details

- Slop Code Bench: A benchmark designed to test long-running agents performing a series of tasks in an unknown order, requiring built-in incremental capability. Agents show significantly lower scores on this type of task compared to simple single-task completion (00:15:24).
- Skills and Prompting: A 'skill' is a defined set of instructions used to guide agents. The effectiveness depends not only on the skill but also on how well the agent reads it; poor disclosure in the prompt can cause agents to ignore critical instructions (00:21:36).
- Code Quality Analysis: Advanced analytics techniques include analyzing agent logs for confusion cycles, running complexity analysis on the codebase, and performing mutation analysis on test suites to identify gaps in coverage or redundant tests (00:23:48).
- Code Governance Models: Implementing a 'risk ladder' for code review, ranging from 'free for all' research codebases to highly controlled areas requiring mandatory human sign-off, is crucial for managing agent-generated code (00:16:57).

## Practical implications

- Shift focus from writing features to defining and enforcing system-level invariants (e.g., naming conventions, data flow rules).
- Integrate deterministic checks (verifiers/linters) into CI pipelines that agents must pass.
- Treat the entire development process as a data source: analyze agent logs, PR comments, and code complexity metrics to identify systemic weaknesses.
- Implement granular risk policies for different parts of the codebase to control merge authority.

## Topics

Build Engineering, AI Development, DevOps, Code Review Automation, Software Architecture, CI/CD Pipelines, Tessl CLI, Tessl: Software Factory

Source: https://www.youtube.com/watch?v=g_iDPqIRvfI