Parameter Golf with AutoResearch — Vayum Arora, Zhengyao Jiang, Dixing Xu & Dhruv Srikanth, Weco AI
The presentation details 'AutoResearch,' an advanced agent-driven loop that automates the process of improving a language model's performance by iteratively proposing code changes and evaluating them. The system was demonstrated using 'Mini Parameter Golf' (Mini PG), a scaled-down version of the original Parameter Golf challenge. The core focus is on designing robust, fast, and inspectable evaluation protocols, enabling agents to perform complex, constrained optimization tasks. The system's advanced features include tree-based search, observability, and steerability, allowing human domain expertise to guide the automated search process.
Key takeaways
-
AutoResearch Mechanism
0:03
AutoResearch operates as a try-and-error loop: an agent modifies a source file, runs the code against a fixed evaluation script, and keeps only the changes that improve the single scalar metric (e.g., accuracy, speed, cost). This process can be applied across various domains, from model training to backend engineering (3:35).
-
Mini Parameter Golf (Mini PG)
0:08
Mini PG scales down the complex Parameter Golf challenge by using a tiny dataset and a quick evaluation, making the auto-research loop visible and inspectable in a workshop setting (8:30).
-
Designing Robust Evaluations
0:10
For auto-research to be useful, the evaluation must be fast, cheap, have low noise, and be robust against 'reward hacking.' The system must output a single scalar metric, and time/compute bounds must be strictly enforced (10:30).
-
Advanced Search Control
0:10
Production-grade auto-research requires three features: 1) Tree-based search (treating the problem as an experiment graph), 2) Observability (seeing the exact code diff and metric change for every node), and 3) Steerability (allowing human users to inject ideas and guide the search when the agent plateaus) (10:30).