# Better AI Models Won't Fix Your Engineering Team — Nicholas Arcolano, Jellyfish

## Executive summary

The speaker argues that while AI models are advancing rapidly, the model itself accounts for less than 1% of the variance in engineering output (measured by merged PRs per engineer). Based on analysis of data from 50,000 engineers, the primary drivers of high throughput are the individual engineer's skills, the company/team environment, and the individual's inherent ability. Leaders are advised to focus on improving people, skills, and organizational processes rather than solely investing in the latest AI tool.

## Key takeaways

- Model Improvement is Minimal: The tool (AI model choice/switching) explains less than 1% of the variation in PR throughput, suggesting that model choice is not the primary bottleneck. (Timestamp: 07:14)
- Skill Use is a Major Lever: Skill use accounts for 3% of the variance, and high skill use (e.g., 30+ runs of using skills per engineer per week) can lead to a 30% bump in PR throughput, holding other factors constant. (Timestamp: 08:07)
- Individual Ability is the Largest Factor: The individual engineer is the largest predictor, accounting for the majority of the statistical model's variance (three quarters). The spread between the bottom 10% and top 10% of engineers is 3.5x wider than for companies. (Timestamp: 12:22)
- Focus on Controllable Factors: Leaders should focus on the environment and people (company, team, skills) because these are the factors that can be controlled, unlike the rapid advancement of external tools. (Timestamp: 14:50)

## Technical details

- Data Source and Scope: Analysis uses data from approximately 50,000 engineers across 447 companies, tracking AI coding tool and agent telemetry alongside engineering workflow data. (Timestamp: 02:59)
- Throughput Metric: The primary metric used for analysis is the change in merged PRs per engineer per week. (Timestamp: 02:59)
- Token Usage: Increased token usage (consumption) shows a positive correlation with throughput, exhibiting diminishing returns, where the top 10% of consumers can double their output compared to the median, but at a 10x cost. (Timestamp: 10:00)
- AI Coding Tools: The analysis compares various tools and modes, including switching between Cursor and Claude Code, or moving from autocomplete mode to agentic mode, finding that these switches do not statistically explain meaningful changes in PR throughput. (Timestamp: 07:14)

## Practical implications

- Leaders should prioritize improving team skills, optimizing the engineering environment, and fostering strong company culture, as these factors have a greater impact on throughput than model choice.
- For engineering teams, focusing on structured skill development (e.g., advanced use of AI tools/harnesses) yields a measurable return on investment.
- While increased token usage correlates with higher output, teams must be aware of the diminishing returns curve to optimize cost efficiency.

## Topics

AI Engineering, Software Development Workflow, Developer Productivity, Data Science, Leadership Strategy, Jellyfish AI engineering trends, AIE CODE summit

Source: https://www.youtube.com/watch?v=Ve2bAarPA2I