# Training Agents 4: From reward functions to environments.

## Executive summary

This session details the evolution of agent training from simple functional reward signals to complex, stateful environments. The core concept is the `reset()/step()` contract, which allows agents to interact with a simulated world (the environment) over a sequence of actions. The discussion covers the OpenM framework, which standardizes environment definition, and its integration with TRL (Transformer Reinforcement Learning) using isolated compute environments like Hugging Face sandboxes. This enables training sophisticated agents, such as coding agents (OpenCode), on complex, multi-step tasks while maintaining reproducibility and isolation.

## Key takeaways

- The Shift to Stateful Environments: For agents performing sequences of actions (e.g., tool calls, file edits), the reward signal must come from the environment's state after an action, rather than being a simple Python function evaluated once. This requires adopting the standard `reset()/step()` contract, moving from sparse signals to continuous interaction loops.
- The OpenM Ecosystem: OpenM provides a standardized, containerized way to define any task as an environment. It packages the task, the runtime (compute backend), and the grading components (verifiers, rewards) into a single, shareable unit that can be deployed on various platforms (e.g., Hugging Face Spaces, Kubernetes).
- Advanced Agentic Training Loops: Training can be managed by two models: the 'White Box' (where the training framework owns the loop) and the 'Black Box' (where the agent/harness owns the loop). The latter requires a 'capture proxy' to intercept agent calls and send the resulting rollout graph back to the trainer for policy updates.
- Reproducibility and Isolation: Environments are designed to be self-contained applications, often deployed in isolated compute environments (like HF sandboxes). This ensures that training runs are reproducible and prevents the agent from accessing or manipulating the verifiers or task description to 'hack' the reward signal.

## Technical details

- RL Environment Contract: The fundamental contract is `reset()` (to define a new world state) and `step()` (to move through the world based on an action). The observation returned must include both the environment state and the reward signal for the training framework (TRL) to use.
- OpenM Workflow: OpenM environments can be managed via a CLI with commands: `init` (to create boilerplate), `push` (to publish to the Hub), `pull` (to retrieve), and `fork` (to create variations). This allows for structured environment development and sharing.
- Training Integration: The training process uses TRL (Transformer Reinforcement Learning) and integrates with OpenM via an `environment_factory`. This allows the trainer to instantiate and manage multiple, complex environments (e.g., OpenCode) in parallel using isolated compute resources like HF sandboxes.
- OpenCode Demo: A practical example demonstrated training a small model (3B parameters) on OpenCode problems using a set of test cases. The reward signal is the fraction of hidden test cases passed, showing a rapid increase from 0 to 1 over 10 steps.

## Practical implications

- Build engineers can treat any piece of software or system (e.g., a microservice, a complex workflow) as a potential RL environment, defining its state, actions, and reward logic.
- The OpenM framework provides a standardized, containerized method for defining and sharing these environments, making them highly portable across different compute backends.
- The concept of using isolated sandboxes (HF sandboxes) for training ensures that the training process is robust, reproducible, and cannot be compromised by the agent attempting to 'hack' the reward signal.
- Environments can be used for evaluation (inference) before training, providing a critical baseline for model capability against a defined task.

## Topics

Reinforcement Learning (RL), Agentic Systems, Build Engineering, Containerization, Hugging Face Ecosystem, LLM Training, OpenM, Hugging Face Hub, TRL (Transformer Reinforcement Learning), OpenCode

Source: https://www.youtube.com/watch?v=nJV3yUuz6DU