# AI Writes the Code. So What's Left for Engineers? - Laurie Voss, Arize AI

## Executive summary

Laurie Voss argues that AI agents have made the act of writing code nearly free, causing the software development bottleneck to shift away from coding. The bottleneck initially moved to code review, but the ultimate constraint is the 'essential complexity'—the difficulty of defining what the user actually needs and what 'good' means. The job of the future is not the 'AI Engineer,' but the 'Product Engineer,' who bridges the gap between business needs and technical specifications. Voss advises that engineers must focus on defining specs, rubrics, and evaluation criteria (evals) to guide agents, rather than focusing on writing code.

## Key takeaways

- The Bottleneck Shift: Agents have made writing code 'basically free.' The bottleneck has moved from coding speed to code review. This pattern will continue, moving through operations and security until it hits a constraint that agents cannot solve.
- Code Review Limitations: While developers using autonomous coding agents write 741% more code, they ship only 30% faster. This inefficiency is attributed to human limitations in code review (e.g., Cisco's finding that human parsing ability falls off after ~450 lines of code per hour).
- The Essential Complexity: The most difficult part of software development is not the code itself, but the 'essential complexity'—the process of figuring out what the user actually needs and defining what 'good' means. This knowledge resides in the human mind, not in training data.
- The Product Engineer Role: The new critical role is the Product Engineer (or Forward Deployed Engineer). This individual's job is to define requirements and specifications, allowing agents to handle the actual coding and implementation.

## Technical details

- Code Output Metrics: Developers using autonomous coding agents write 741% more code, but the resulting shipping speed increase is only 30%. This disparity indicates a major bottleneck downstream from the coding process.
- Code Review Benchmarking: Standard benchmarks (like Claude Fable 5) test if code passes tests (80% pass rate). However, a more accurate benchmark (like Cognition's) asks if a maintainer would merge the PR, which results in a lower pass rate (54%) because it accounts for external context, code style, and history.
- The Theory of Constraints: Every system has one bottleneck (Eliyahu Goldratt). Improving one area (like coding) does not improve the system unless that area was the bottleneck. The next bottleneck will always appear after the current one is solved.
- Modern Development Process: The process is shifting from 'programmer' (writing code) to 'product engineer' (defining the problem). The goal is to put knowledge that exists only in people's heads into specs, rubrics, and evaluation criteria (evals) for the agents to follow.

## Practical implications

- Focus on defining the problem space: Spend time creating detailed specifications, rubrics, and evaluation criteria (evals) that capture institutional knowledge.
- Measure actual shipping output, not lines of code. Track metrics like 'how often reviewer comments get acted on' or the gap between 'tests passed' and 'maintainer would merge.'
- Build systems that automatically verify the 'high blast radius' code, delegating everything that is cheap to verify to agents.
- Engineering leaders must intentionally train the next generation of engineers in product discovery and specification writing, rather than just coding.

## Topics

AI Agents, Software Architecture, Product Management, Build Engineering, Software Development Lifecycle (SDLC), Arize AI, npm, The Goal, Theory of Constraints, The Mythical Man-Month, GitHub Copilot

Source: https://www.youtube.com/watch?v=kspZP5BgkEU