# How To Turn Evals Into A Better Model

## Executive summary

This session details advanced strategies for improving Large Language Model (LLM) performance, arguing that optimizing the evaluation environment (Evals) is often more impactful than immediate fine-tuning. The core components of an eval are the Task Set, the Harness, and the Scoring Function. Furthermore, it provides a deep dive into Reinforcement Learning (RL), outlining its architecture—including inference engines (VLM, SGLang), orchestrators, and trainers—and warning about common pitfalls like reward hacking.

## Key takeaways

- Prioritize Eval Improvement Over Fine-Tuning: Before fine-tuning a model, thoroughly audit the evaluation setup. Improvements can often be found by adjusting sampling parameters (e.g., using temperature > 0), swapping harnesses (like Pi for open-source control), or increasing resource allocation/timeouts. [0:08:19]
- Understand the Three Parts of an Eval: Every evaluation consists of three parts: the Task Set (data, prompts, tools); the Harness (the program loop driving LLM interaction with an environment); and the Scoring Function/Reward Function (which can be deterministic or use a judge LLM). [0:00:55]
- Reinforcement Learning (RL) is for System Improvement: RL is a powerful, advanced training algorithm used to improve model capabilities on specific tasks by learning from trial and error. It should be considered the last step after optimizing the eval environment. [0:13:24]
- Beware of Reward Hacking: RL is highly sensitive to weak or poorly designed evals. Models may learn to optimize for a proxy metric (reward hacking) rather than solving the true underlying task, necessitating careful evaluation design and red teaming. [0:22:24]

## Technical details

- Eval Components: An eval requires three parts: 1) Task Set (data, prompts, tools); 2) Harness (the program loop driving the LLM interaction with an environment, e.g., Codeex, Cloud Code, Pi); and 3) Scoring Function/Reward Function (deterministic comparison or judge LLMs). [0:00:55]
- Hidden Variables in Evals: Results can be influenced by factors outside the model itself, including the Inference Engine/API used (e.g., calling a closed model via Bedrock vs. first-party API), the underlying hardware/sandbox resources, and task timeouts. [0:06:07]
- RL Training Architecture: An RL training system involves three key components: an Inference Engine (VLM, SGLang) for generating rollouts; an Orchestrator to manage the process and communicate requests; and a Trainer (using algorithms like PPO or GRPO) that constructs gradients based on rewards. [0:17:56]
- Sampling Parameters Best Practices: Avoid setting temperature to zero, as this is generally incorrect for modern LLMs. Using a temperature of one is often better than zero, though model makers' recommendations should always be followed. Also, ensure high max tokens/max turns settings to prevent truncation. [0:09:30]

## Practical implications

- Always audit the infrastructure (harness, API, hardware) before assuming model failure.
- When designing an eval, focus on making the scoring function as verifiable and rule-based as possible to mitigate reward hacking risks.
- Start optimization efforts by adjusting sampling parameters (temperature, max tokens) rather than jumping straight to fine-tuning or RL.
- Use open-source tools like Pi for maximum control over your evaluation infrastructure.

## Topics

Large Language Models, ML Ops, Reinforcement Learning, Evaluation Design, AI Infrastructure, Pi (open-source harness), Codeex / Cloud Code, VLM / SGLang, Primalog/Prime Intellect

Source: https://www.youtube.com/watch?v=P7os3s-HMAw