Better AI Models Won't Fix Your Engineering Team — Nicholas Arcolano, Jellyfish
Summary
The speaker argues that while AI models are advancing rapidly, the model itself accounts for less than 1% of the variance in engineering output (measured by merged PRs per engineer). Based on analysis of data from 50,000 engineers, the primary drivers of high throughput are the individual engineer's skills, the company/team environment, and the individual's inherent ability. Leaders are advised to focus on improving people, skills, and organizational processes rather than solely investing in the latest AI tool.
Key takeaways
-
Model Improvement is Minimal
11:54
The tool (AI model choice/switching) explains less than 1% of the variation in PR throughput, suggesting that model choice is not the primary bottleneck. (Timestamp: 07:14)
-
Skill Use is a Major Lever
13:27
Skill use accounts for 3% of the variance, and high skill use (e.g., 30+ runs of using skills per engineer per week) can lead to a 30% bump in PR throughput, holding other factors constant. (Timestamp: 08:07)
-
Individual Ability is the Largest Factor
The individual engineer is the largest predictor, accounting for the majority of the statistical model's variance (three quarters). The spread between the bottom 10% and top 10% of engineers is 3.5x wider than for companies. (Timestamp: 12:22)
-
Focus on Controllable Factors
Leaders should focus on the environment and people (company, team, skills) because these are the factors that can be controlled, unlike the rapid advancement of external tools. (Timestamp: 14:50)
Technical details
-
Data Source and Scope
259s
Analysis uses data from approximately 50,000 engineers across 447 companies, tracking AI coding tool and agent telemetry alongside engineering workflow data. (Timestamp: 02:59)
-
Throughput Metric
259s
The primary metric used for analysis is the change in merged PRs per engineer per week. (Timestamp: 02:59)
-
Token Usage
600s
Increased token usage (consumption) shows a positive correlation with throughput, exhibiting diminishing returns, where the top 10% of consumers can double their output compared to the median, but at a 10x cost. (Timestamp: 10:00)
-
AI Coding Tools
The analysis compares various tools and modes, including switching between Cursor and Claude Code, or moving from autocomplete mode to agentic mode, finding that these switches do not statistically explain meaningful changes in PR throughput. (Timestamp: 07:14)
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.