Harness Engineering is not Enough: Why Software Factories Fail — Dex Horthy, HumanLayer
The video argues that current efforts in 'harness engineering' and increasing tokens are insufficient for building reliable AI software factories because they fail to address fundamental model training shortcomings. The core problem is maintaining codebase quality over time (maintainability), which current reward functions do not penalize. To move forward safely, the speaker advocates returning to rigorous human-led upfront planning: Product Review $\rightarrow$ System Architecture $\rightarrow$ Program Design (down to types and call graphs) $\rightarrow$ Vertical Slices.
Key takeaways
-
The Failure of 'Lights Off' Factories
12:10
Attempting to run a software factory with no human code review ('lights off') leads to failures, even for advanced agents. The issue is not scale or prompting, but a fundamental model training limitation.
-
Model Training Flaw (The Maintainability Gap)
17:12
Current coding models are primarily trained on passing tests and solving one-off problems. Their reward signal does not penalize poor program design or the erosion of codebase maintainability, meaning they get better at passing tests but worse at keeping large systems stable.
-
The Path Forward: Structured Planning
To move faster safely, engineers must re-emphasize upfront planning steps: Product Review (desired behavior/mockups), System Architecture (component contracts/data models), Program Design (types and call graphs), and Vertical Slices (implementation order).
