How To Turn Evals Into A Better Model
This session details advanced strategies for improving Large Language Model (LLM) performance, arguing that optimizing the evaluation environment (Evals) is often more impactful than immediate fine-tuning. The core components of an eval are the Task Set, the Harness, and the Scoring Function. Furthermore, it provides a deep dive into Reinforcement Learning (RL), outlining its architecture—including inference engines (VLM, SGLang), orchestrators, and trainers—and warning about common pitfalls like reward hacking.
Key takeaways
-
Prioritize Eval Improvement Over Fine-Tuning
9:39
Before fine-tuning a model, thoroughly audit the evaluation setup. Improvements can often be found by adjusting sampling parameters (e.g., using temperature > 0), swapping harnesses (like Pi for open-source control), or increasing resource allocation/timeouts. [0:08:19]
-
Understand the Three Parts of an Eval
5:15
Every evaluation consists of three parts: the Task Set (data, prompts, tools); the Harness (the program loop driving LLM interaction with an environment); and the Scoring Function/Reward Function (which can be deterministic or use a judge LLM). [0:00:55]
-
Reinforcement Learning (RL) is for System Improvement
13:24
RL is a powerful, advanced training algorithm used to improve model capabilities on specific tasks by learning from trial and error. It should be considered the last step after optimizing the eval environment. [0:13:24]
-
Beware of Reward Hacking
22:24
RL is highly sensitive to weak or poorly designed evals. Models may learn to optimize for a proxy metric (reward hacking) rather than solving the true underlying task, necessitating careful evaluation design and red teaming. [0:22:24]