How AI agents reproduced ICML 2026 papers
Summary
The ICML 2026 Agents Reproduction Challenge was a large-scale community effort involving over 1,200 participants and AI agents attempting to reproduce claims from accepted machine learning papers. The initiative demonstrated the potential for automated reproducibility testing in academic research, finding that while a majority of papers were reproducible (some fully, some via smaller scale experiments), significant flaws were also identified. Key technical takeaways include the use of specialized tools like `tracko` and Hugging Face infrastructure to create fully auditable, machine-readable log books for every reproduction attempt.
Key takeaways
-
Scale of Reproduction Effort
4:18
The challenge involved 1,200+ participants attempting to reproduce claims from a subset of ICML 2026 papers. A total of 2,200 unique papers were attempted, resulting in approximately 35,000 different claims being judged (Timestamp: ~4:18).
-
Reproducibility Success Rate
12:34
A majority of the papers looked at were reproducible. Specifically, over 2,000 papers had at least one major claim independently verified (Timestamp: ~6:34).
-
Identification of Flaws and Contested Claims
13:10
The community found that about 23% of papers could not be fully reproduced as claimed, leading to at least 496 contested or falsified claims. Furthermore, 49 papers were almost fully falsified (Timestamp: ~8:15).
-
Best Practices in Agent Use
15:42
The 'Best Human in the Loop' award highlighted that effective reproduction requires human intervention to guide agents, especially when evaluating qualitative results (e.g., building a UI to compare quantized images) (Timestamp: ~10:35).
Technical details
-
Experiment Tracking and Auditing
320s
The process utilized `tracko`, an open-source experiment tracking library similar to Weights & Biases or Neptune ML. This system generates 'log books' that log arbitrary markdown/code cells, inputs, outputs, and created artifacts (e.g., CSV files, model checkpoints) for full reproducibility (Timestamp: ~5:20).
-
Agent Workflow Logging
415s
The system supports logging not only the code and outputs but also 'agent traces'—the raw, human-readable back-and-forth between user messages, agent messages, and tool calls. These logs are pushed to Hugging Face Spaces/DataSets (Timestamp: ~6:30).
-
Compute Infrastructure
1080s
The results and log books were hosted on Hugging Face, utilizing cloud storage optimized for AI/ML workloads. Participants can use the credits earned to run jobs via 'Hugging Face Jobs' or serverless inference (Timestamp: ~10:50).
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.