How To Turn Evals Into A Better Model
Summary
This session details advanced strategies for improving Large Language Model (LLM) performance, arguing that optimizing the evaluation environment (Evals) is often more impactful than immediate fine-tuning. The core components of an eval are the Task Set, the Harness, and the Scoring Function. Furthermore, it provides a deep dive into Reinforcement Learning (RL), outlining its architecture—including inference engines (VLM, SGLang), orchestrators, and trainers—and warning about common pitfalls like reward hacking.
Key takeaways
-
Prioritize Eval Improvement Over Fine-Tuning
9:39
Before fine-tuning a model, thoroughly audit the evaluation setup. Improvements can often be found by adjusting sampling parameters (e.g., using temperature > 0), swapping harnesses (like Pi for open-source control), or increasing resource allocation/timeouts. [0:08:19]
-
Understand the Three Parts of an Eval
5:15
Every evaluation consists of three parts: the Task Set (data, prompts, tools); the Harness (the program loop driving LLM interaction with an environment); and the Scoring Function/Reward Function (which can be deterministic or use a judge LLM). [0:00:55]
-
Reinforcement Learning (RL) is for System Improvement
13:24
RL is a powerful, advanced training algorithm used to improve model capabilities on specific tasks by learning from trial and error. It should be considered the last step after optimizing the eval environment. [0:13:24]
-
Beware of Reward Hacking
22:24
RL is highly sensitive to weak or poorly designed evals. Models may learn to optimize for a proxy metric (reward hacking) rather than solving the true underlying task, necessitating careful evaluation design and red teaming. [0:22:24]
Technical details
-
Eval Components
315s
An eval requires three parts: 1) Task Set (data, prompts, tools); 2) Harness (the program loop driving the LLM interaction with an environment, e.g., Codeex, Cloud Code, Pi); and 3) Scoring Function/Reward Function (deterministic comparison or judge LLMs). [0:00:55]
-
Hidden Variables in Evals
367s
Results can be influenced by factors outside the model itself, including the Inference Engine/API used (e.g., calling a closed model via Bedrock vs. first-party API), the underlying hardware/sandbox resources, and task timeouts. [0:06:07]
-
RL Training Architecture
1076s
An RL training system involves three key components: an Inference Engine (VLM, SGLang) for generating rollouts; an Orchestrator to manage the process and communicate requests; and a Trainer (using algorithms like PPO or GRPO) that constructs gradients based on rewards. [0:17:56]
-
Sampling Parameters Best Practices
570s
Avoid setting temperature to zero, as this is generally incorrect for modern LLMs. Using a temperature of one is often better than zero, though model makers' recommendations should always be followed. Also, ensure high max tokens/max turns settings to prevent truncation. [0:09:30]
Mentioned resources
- Pi (open-source harness)
- Codeex / Cloud Code
- VLM / SGLang
- Primalog/Prime Intellect
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.