The Video Signal

Topic

LLM Training

All digests tagged LLM Training

Training Agents 4: From reward functions to environments. thumbnail

· 1:13:40

Training Agents 4: From reward functions to environments.

This session details the evolution of agent training from simple functional reward signals to complex, stateful environments. The core concept is the `reset()/step()` contract, which allows agents to interact with a simulated world (the environment) over a sequence of actions. The discussion covers the OpenM framework, which standardizes environment definition, and its integration with TRL (Transformer Reinforcement Learning) using isolated compute environments like Hugging Face sandboxes. This enables training sophisticated agents, such as coding agents (OpenCode), on complex, multi-step tasks while maintaining reproducibility and isolation.

Key takeaways

  1. The Shift to Stateful Environments 1:42

    For agents performing sequences of actions (e.g., tool calls, file edits), the reward signal must come from the environment's state after an action, rather than being a simple Python function evaluated once. This requires adopting the standard `reset()/step()` contract, moving from sparse signals to continuous interaction loops.

  2. The OpenM Ecosystem 5:30

    OpenM provides a standardized, containerized way to define any task as an environment. It packages the task, the runtime (compute backend), and the grading components (verifiers, rewards) into a single, shareable unit that can be deployed on various platforms (e.g., Hugging Face Spaces, Kubernetes).

  3. Advanced Agentic Training Loops 9:10

    Training can be managed by two models: the 'White Box' (where the training framework owns the loop) and the 'Black Box' (where the agent/harness owns the loop). The latter requires a 'capture proxy' to intercept agent calls and send the resulting rollout graph back to the trainer for policy updates.

  4. Reproducibility and Isolation 7:30

    Environments are designed to be self-contained applications, often deployed in isolated compute environments (like HF sandboxes). This ensures that training runs are reproducible and prevents the agent from accessing or manipulating the verifiers or task description to 'hack' the reward signal.

Watch on YouTube Full article

The Base Model Is Dead — Varun Singh, Arcee AI thumbnail

· 17:45

The Base Model Is Dead — Varun Singh, Arcee AI

The traditional paradigm of viewing a base language model as merely a reflection of the entire human web is obsolete. As LLMs advance toward complex reasoning and agentic behavior (e.g., interacting with software environments), the training focus shifts from raw web text to incorporating instruction data, synthetic reasoning traces, and post-training techniques earlier into the pre-training phase. This requires careful management of data mixes, load balancing coefficients (especially in MOEs), and establishing stable representations early on.

Key takeaways

  1. The Decline of Raw Web Text 2:10

    Historically, models like GPT-3 relied heavily on raw web scrapes (e.g., Common Crawl/WebText-2), which constituted up to 85% of the training data. Modern recipes show a significant decrease in web text's proportion, indicating that its value is diminishing relative to code and structured reasoning abilities.

  2. Shift from Knowledge Prior to Capability Prior 8:00

    The base model's role is changing from accumulating general world knowledge (the 'prior') to carrying the necessary prior for complex Reinforcement Learning (RL) tasks. RL is no longer a mere 'cherry on top,' but a core component that requires the base model to be prepared for advanced composition and reasoning.

  3. Synthetic Data Integration 11:20

    A key trend involves pulling post-training data (like SFT/Q&A chat data) and large-scale synthetic data back into the pre-training phase. This allows models to learn task representations, conversation shapes, and atomic skills from the very beginning.

Watch on YouTube Full article