Parameter Golf with AutoResearch — Vayum Arora, Zhengyao Jiang, Dixing Xu & Dhruv Srikanth, Weco AI
Summary
The presentation details 'AutoResearch,' an advanced agent-driven loop that automates the process of improving a language model's performance by iteratively proposing code changes and evaluating them. The system was demonstrated using 'Mini Parameter Golf' (Mini PG), a scaled-down version of the original Parameter Golf challenge. The core focus is on designing robust, fast, and inspectable evaluation protocols, enabling agents to perform complex, constrained optimization tasks. The system's advanced features include tree-based search, observability, and steerability, allowing human domain expertise to guide the automated search process.
Key takeaways
-
AutoResearch Mechanism
0:03
AutoResearch operates as a try-and-error loop: an agent modifies a source file, runs the code against a fixed evaluation script, and keeps only the changes that improve the single scalar metric (e.g., accuracy, speed, cost). This process can be applied across various domains, from model training to backend engineering (3:35).
-
Mini Parameter Golf (Mini PG)
0:08
Mini PG scales down the complex Parameter Golf challenge by using a tiny dataset and a quick evaluation, making the auto-research loop visible and inspectable in a workshop setting (8:30).
-
Designing Robust Evaluations
0:10
For auto-research to be useful, the evaluation must be fast, cheap, have low noise, and be robust against 'reward hacking.' The system must output a single scalar metric, and time/compute bounds must be strictly enforced (10:30).
-
Advanced Search Control
0:10
Production-grade auto-research requires three features: 1) Tree-based search (treating the problem as an experiment graph), 2) Observability (seeing the exact code diff and metric change for every node), and 3) Steerability (allowing human users to inject ideas and guide the search when the agent plateaus) (10:30).
Technical details
-
AutoResearch Components
11s
The system requires defining a Source File (the search space, e.g., `solution.py`), a fixed Evaluation Command (the bash command that runs the test and outputs a score), and a measurable Goal/Metric (e.g., minimizing `validation BPB`) (11:30).
-
Aiden System
5s
The team's agent, Aiden, is a self-improving multi-agent system that automated the research loop, achieving record contributions in the OpenAI Parameter Golf challenge (5:39).
-
Flappy Bird Competition
14s
The final exercise adapted auto-research to optimize a Flappy Bird controller. The metric was the average score across four different worlds (normal, moving gap, wind, variable), emphasizing the need for robust, generalizable solutions and using a hidden validation set (14:30).
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.