# How AI agents reproduced ICML 2026 papers

## Executive summary

The ICML 2026 Agents Reproduction Challenge was a large-scale community effort involving over 1,200 participants and AI agents attempting to reproduce claims from accepted machine learning papers. The initiative demonstrated the potential for automated reproducibility testing in academic research, finding that while a majority of papers were reproducible (some fully, some via smaller scale experiments), significant flaws were also identified. Key technical takeaways include the use of specialized tools like `tracko` and Hugging Face infrastructure to create fully auditable, machine-readable log books for every reproduction attempt.

## Key takeaways

- Scale of Reproduction Effort: The challenge involved 1,200+ participants attempting to reproduce claims from a subset of ICML 2026 papers. A total of 2,200 unique papers were attempted, resulting in approximately 35,000 different claims being judged (Timestamp: ~4:18).
- Reproducibility Success Rate: A majority of the papers looked at were reproducible. Specifically, over 2,000 papers had at least one major claim independently verified (Timestamp: ~6:34).
- Identification of Flaws and Contested Claims: The community found that about 23% of papers could not be fully reproduced as claimed, leading to at least 496 contested or falsified claims. Furthermore, 49 papers were almost fully falsified (Timestamp: ~8:15).
- Best Practices in Agent Use: The 'Best Human in the Loop' award highlighted that effective reproduction requires human intervention to guide agents, especially when evaluating qualitative results (e.g., building a UI to compare quantized images) (Timestamp: ~10:35).

## Technical details

- Experiment Tracking and Auditing: The process utilized `tracko`, an open-source experiment tracking library similar to Weights & Biases or Neptune ML. This system generates 'log books' that log arbitrary markdown/code cells, inputs, outputs, and created artifacts (e.g., CSV files, model checkpoints) for full reproducibility (Timestamp: ~5:20).
- Agent Workflow Logging: The system supports logging not only the code and outputs but also 'agent traces'—the raw, human-readable back-and-forth between user messages, agent messages, and tool calls. These logs are pushed to Hugging Face Spaces/DataSets (Timestamp: ~6:30).
- Compute Infrastructure: The results and log books were hosted on Hugging Face, utilizing cloud storage optimized for AI/ML workloads. Participants can use the credits earned to run jobs via 'Hugging Face Jobs' or serverless inference (Timestamp: ~10:50).

## Practical implications

- Establishes a new standard for reproducibility in ML research by providing fully auditable log books that track code execution and all resulting artifacts.
- Highlights the necessity of 'Human-in-the-Loop' validation, suggesting that automated agent reproduction alone is insufficient for complex evaluations (e.g., visual quality assessment).
- Provides a framework for build engineers to integrate rigorous, multi-agent testing into research CI/CD pipelines, moving beyond simple code execution checks.

## Topics

AI Agents, Machine Learning Reproducibility, Experiment Tracking, Build Engineering, Large Language Models (LLMs), ICML 2026 Agents Reproduction Challenge Website, tracko

Source: https://www.youtube.com/watch?v=8qpEDpNhzWY