Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs
Summary
The talk analyzes the necessity of undertaking a major codebase refactor—specifically collapsing over ten legacy repositories into a monorepo—despite rapid advancements in AI coding agents. The speaker argues that while modern LLMs (like Opus 4.8) significantly accelerate tasks compared to older models (o3), the business value gained from proactively addressing technical debt and establishing robust development patterns outweighs waiting for perfect AI tooling. Key findings include benchmarking model performance on refactoring tasks, noting that Sonnet 4.6 required one iteration while Opus 4.8 achieved near single-pass completion. The speaker also cautions against relying solely on LLMs to complete complex tasks without proper guardrails and validation.
Key takeaways
-
Refactoring is necessary despite AI progress
The refactor was worthwhile because it improved business metrics (commit velocity, time-to-market) by establishing clean patterns, even if models continue to improve.
-
LLM performance benchmarks show rapid improvement
9:30
Refactoring tasks that took three hours of back and forth with o3 could be accomplished in roughly one-fifth the time using modern models like Sonnet 4.6 (one iteration) or Opus 4.8 (near single pass).
-
Monorepos simplify development flow
Consolidating ten separate repositories into a monorepo allowed almost every developer to contribute, even outside their area of expertise, significantly boosting commit velocity and collaboration.
Technical details
-
AI Model Benchmarking (Refactoring)
570s
The speaker benchmarked refactor tasks using o3 vs. modern models: o3 took three hours and resulted in ten major mistakes. Sonnet 4.6 required one extra iteration, while Opus 4.8 achieved the task in essentially one pass. This represents a significant reduction in manual effort and time.
-
LLM Limitations (GPT-5.5)
360s
When given a complex refactoring goal, GPT 5.5 extra high completed the task quickly (10 minutes 22 seconds) but only wrote scaffolding code (2,000 lines), admitting that it had not added necessary deployment or bootstrap commands.
-
Technical Debt and Code Quality
240s
The speaker advises viewing technical debt rigorously: introducing complexity can quickly outrun the potential Return on Investment (ROI). Maintaining code quality is critical, even when adopting AI-native development practices.
-
Agentic Workflow Development
300s
Modern tooling allows for building agentic workflows that involve deep research, matching requirements to problem statements, creating sub-agents for criteria evaluation, and finally building Proofs of Concept (PoCs), accelerating processes previously requiring manual Confluence documentation.
Mentioned resources
- WiseDocs
- Sonnet 4.6
- Opus 4.8
- GPT 5.5 extra high
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.