We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect
Summary
The presentation details an experiment where AI agents (Claude Code and Codex) competed in an 'Optimizer Speedrun' to achieve a new record for training a GPT-2-level model. While the agents successfully beat the human record, the speaker's key finding is that they achieved this by combining existing ideas rather than inventing novel optimizers or mechanisms. To advance AI research beyond mere evaluation, the speaker proposes an 'AlphaEvolve-style discovery loop' that integrates multi-agent interaction, quality feedback, and scaling elements.
Key takeaways
-
AI Agents Beat Human Records in Speedrun
17:01
In the Optimizer Speedrun, both Codex and Claude Code significantly outperformed the human record, achieving a new best record for training a GPT-2-level model. (10:21)
-
Agents Exhibit Different Behaviors
13:56
Codex was observed to write extensively in its 'scratchpad' (active memory), spawn more sub-agents, and burn more tokens compared to Claude Code, which frequently became idle and stated it could not improve the record. (7:46, 8:36)
-
Lack of Novel Discovery
Despite the impressive results, the models did not invent a new optimizer or mechanism. Instead, they combined existing ideas for small gains, suggesting that current methods are more geared toward evaluation than true discovery. (14:35)
-
Proposed Discovery Loop
The speaker proposes an AlphaEvolve-style multi-agent system that includes generators (LLMs), a reward mechanism (speedrun), a judge (quality feedback), and a scaling element to guide research toward novel breakthroughs. (15:40)
Technical details
-
Automated AI Research Benchmarking
37s
The goal is to test model capabilities in research, addressing the lack of independent, third-party benchmarks for claims like recursive self-improvement (RSI). (0:37)
-
Speedrun Mechanics
312s
The Optimizer Speedrun constrains agents to changing only optimizer-related parameters (e.g., Adam to Shampoo), making it a method-finding benchmark rather than just a program optimization challenge. (3:12)
-
Computational Environment
652s
The agents operate on a Slurm cluster using `sbatch` for job submission. Jobs are run using 'preemptable permission,' allowing the model to cancel jobs if the node is needed by others. (6:52)
-
Model Architecture and Agents
652s
The agents used were Codex (GPT 5.5 with XI) and Claude Code (Opus 4.8 with XI). The system utilizes a `goal.md` file to define rules and manage the research process. (6:52)
Mentioned resources
- Prime Intellect
- Claude Code
- Codex
- GPT-2
- AlphaEvolve
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.