# We Let Claude Code and Codex Race Human Researchers — Elie Bakouch, Prime Intellect

## Executive summary

The presentation details an experiment where AI agents (Claude Code and Codex) competed in an 'Optimizer Speedrun' to achieve a new record for training a GPT-2-level model. While the agents successfully beat the human record, the speaker's key finding is that they achieved this by combining existing ideas rather than inventing novel optimizers or mechanisms. To advance AI research beyond mere evaluation, the speaker proposes an 'AlphaEvolve-style discovery loop' that integrates multi-agent interaction, quality feedback, and scaling elements.

## Key takeaways

- AI Agents Beat Human Records in Speedrun: In the Optimizer Speedrun, both Codex and Claude Code significantly outperformed the human record, achieving a new best record for training a GPT-2-level model. (10:21)
- Agents Exhibit Different Behaviors: Codex was observed to write extensively in its 'scratchpad' (active memory), spawn more sub-agents, and burn more tokens compared to Claude Code, which frequently became idle and stated it could not improve the record. (7:46, 8:36)
- Lack of Novel Discovery: Despite the impressive results, the models did not invent a new optimizer or mechanism. Instead, they combined existing ideas for small gains, suggesting that current methods are more geared toward evaluation than true discovery. (14:35)
- Proposed Discovery Loop: The speaker proposes an AlphaEvolve-style multi-agent system that includes generators (LLMs), a reward mechanism (speedrun), a judge (quality feedback), and a scaling element to guide research toward novel breakthroughs. (15:40)

## Technical details

- Automated AI Research Benchmarking: The goal is to test model capabilities in research, addressing the lack of independent, third-party benchmarks for claims like recursive self-improvement (RSI). (0:37)
- Speedrun Mechanics: The Optimizer Speedrun constrains agents to changing only optimizer-related parameters (e.g., Adam to Shampoo), making it a method-finding benchmark rather than just a program optimization challenge. (3:12)
- Computational Environment: The agents operate on a Slurm cluster using `sbatch` for job submission. Jobs are run using 'preemptable permission,' allowing the model to cancel jobs if the node is needed by others. (6:52)
- Model Architecture and Agents: The agents used were Codex (GPT 5.5 with XI) and Claude Code (Opus 4.8 with XI). The system utilizes a `goal.md` file to define rules and manage the research process. (6:52)

## Practical implications

- The need for open, third-party benchmarks to validate claims of advanced AI capabilities like recursive self-improvement.
- The development of structured, multi-agent systems (like the proposed AlphaEvolve loop) that integrate discovery, feedback, and scaling to move beyond simple evaluation.
- The importance of defining multiple, constrained speedruns (e.g., varying objectives or constraints) to maximize research diversity.

## Topics

AI Research, Large Language Models (LLMs), Benchmarking, Build Automation, Computational Systems, Prime Intellect, Claude Code, Codex, GPT-2, AlphaEvolve

Source: https://www.youtube.com/watch?v=oVsEddfhdxc