Agents Write 95% of Our Code. Here's the Catch
Summary
As AI agents assume control over an estimated 95% of code production in advanced software factories, traditional code review processes are insufficient. The talk introduces the role of the 'harness engineer,' a new skill set focused on system-level controls: defining invariants, performing deep analytics on agent logs and PR data, and implementing fine-grained risk/operations policies (like auto-merge ladders). This shift requires engineers to move from writing code features to building robust guardrails that ensure consistency and quality across agent-driven pipelines.
Key takeaways
-
The Paradox of AI Adoption
25:24
While AI coding tool adoption is high, benchmarks are becoming saturated. Concurrently, the number of reported bugs and incidents is rising, indicating that agents may generate code that lacks maintainability or systemic health (00:15:24).
-
The Rise of the Harness Engineer
9:34
Engineering focus must shift from pure feature building to defining and enforcing system invariants. The three critical new skill sets are Systems Thinking, Analytics, and Risk/Operations (00:09:34).
-
Instruction Following Gap in Skills
8:23
Tessl's internal skills benchmark revealed that while agents achieved high task completion rates, they only followed approximately 70% of the total instructions defined within a skill (00:08:22).
-
Systemic Control through Invariants and CI Gates
12:56
Engineers must identify general principles (invariants)—such as design system rules or desired code structure—and encode them into deterministic checks, verifiers, or CI gates to ensure consistency across the codebase (00:12:56).
Technical details
-
Slop Code Bench
1524s
A benchmark designed to test long-running agents performing a series of tasks in an unknown order, requiring built-in incremental capability. Agents show significantly lower scores on this type of task compared to simple single-task completion (00:15:24).
-
Skills and Prompting
1296s
A 'skill' is a defined set of instructions used to guide agents. The effectiveness depends not only on the skill but also on how well the agent reads it; poor disclosure in the prompt can cause agents to ignore critical instructions (00:21:36).
-
Code Quality Analysis
1428s
Advanced analytics techniques include analyzing agent logs for confusion cycles, running complexity analysis on the codebase, and performing mutation analysis on test suites to identify gaps in coverage or redundant tests (00:23:48).
-
Code Governance Models
1017s
Implementing a 'risk ladder' for code review, ranging from 'free for all' research codebases to highly controlled areas requiring mandatory human sign-off, is crucial for managing agent-generated code (00:16:57).
Mentioned resources
- Tessl CLI
- Tessl: Software Factory
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.