Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs
The talk analyzes the necessity of undertaking a major codebase refactor—specifically collapsing over ten legacy repositories into a monorepo—despite rapid advancements in AI coding agents. The speaker argues that while modern LLMs (like Opus 4.8) significantly accelerate tasks compared to older models (o3), the business value gained from proactively addressing technical debt and establishing robust development patterns outweighs waiting for perfect AI tooling. Key findings include benchmarking model performance on refactoring tasks, noting that Sonnet 4.6 required one iteration while Opus 4.8 achieved near single-pass completion. The speaker also cautions against relying solely on LLMs to complete complex tasks without proper guardrails and validation.
Key takeaways
-
Refactoring is necessary despite AI progress
The refactor was worthwhile because it improved business metrics (commit velocity, time-to-market) by establishing clean patterns, even if models continue to improve.
-
LLM performance benchmarks show rapid improvement
9:30
Refactoring tasks that took three hours of back and forth with o3 could be accomplished in roughly one-fifth the time using modern models like Sonnet 4.6 (one iteration) or Opus 4.8 (near single pass).
-
Monorepos simplify development flow
Consolidating ten separate repositories into a monorepo allowed almost every developer to contribute, even outside their area of expertise, significantly boosting commit velocity and collaboration.