# Workshop: Training Agents at the frontier of AI Industry

## Executive summary

This workshop provides an in-depth look at the industrial challenges and advanced methodologies for post-training large language models (LLMs) into functional agents. Speakers cover everything from enterprise system design (latency, de-risking, containment) to advanced RL techniques (SFT, DPO, distillation, GRPO). Key focus areas include building robust, measurable environments, scaling training on massive models (up to 320B parameters), and establishing centralized, vetted resources like OpenEnv Arena to combat issues like reward hacking and answer leakage.

## Key takeaways

- System Design is Paramount for Enterprise Agents: For enterprise use cases, success depends less on the model itself and more on the entire system, including tools, data, network, and wall-clock time. Optimization requires traditional distributed systems engineering techniques like parallelizing API calls, introducing caching layers, and using materialized views to improve latency and accuracy. Containment and least privilege are critical security concerns.
- Distinguishing Rewards from Measurements in Evaluation: A critical distinction must be made between a 'reward' (used during training to guide the model) and a 'measurement' (used during evaluation to assess reliability). Training environments should be designed to produce verifiable reward signals, while evaluation benchmarks must be rigorously assessed for 'fitness to purpose' to ensure the score predicts real-world performance.
- Scaling Agent RL with Fully Asynchronous Infrastructure: Training agents requires handling multi-step interactions (trajectories) rather than single responses. Frameworks like Miles support this by enabling 'fully async RL,' where inference and training run on separate GPU groups. This prevents slow episodes from idling the training GPUs, which is crucial when scaling to large models (e.g., GLM 5.3, 320B parameters).
- The Need for Vetted, Scalable Environments: The complexity of post-training requires centralized, audited resources. OpenEnv Arena aims to vet environments for trainability, checking for failure modes like 'answer leakage' and 'reward hacking.' The process involves filtering vast numbers of potential environments to ensure they provide a reliable, non-trivial training signal.

## Technical details

- Post-Training Techniques: Methods include Supervised Fine-Tuning (SFT) using good traces, Direct Preference Optimization (DPO) using pairwise preferences (e.g., thumbs up/down data), and distillation (e.g., reverse KL distillation). Reinforcement Learning (RL) uses policy gradient methods like GRPO, which must be carefully implemented to prevent divergence.
- Agentic System Architecture: Modern agentic systems are highly modular, often relying on conditional logic and routing between specialized, smaller decision models (e.g., using a small LM3 model for routing, a small vision model for vision decisions, and a separate reasoning model for planning). This approach is preferred over using a single large model for all tasks.
- MLOps and Development Practices: The industry is shifting from traditional ML development to MLOps, requiring robust CI/CD practices for statistical models. This includes maintaining comprehensive regression suites, implementing must-pass tests (for conformity), and adopting a modular, 'pod' structure that integrates Domain Experts (FDEs), ML Engineers, and IT Security teams.
- Training Data Integrity: When building benchmarks, it is crucial to ensure the task is 'gradable' and has clear right/wrong outcomes. The process must account for potential data flaws, such as 'answer leakage' (where the answer is present in the environment's source code/history) or 'false positives/negatives' in the reward signal.

## Practical implications

- Adopt a modular, multi-agent architecture rather than relying on a single monolithic model.
- Treat the evaluation environment as a product that requires continuous maintenance, updating, and vetting, rather than a static benchmark.
- Implement MLOps practices (CI/CD, regression testing, canary deployments) tailored for statistical models, recognizing that the failure modes are different from traditional software.
- Focus on improving the 'harness' and the surrounding system infrastructure (data pipelines, API calls, caching) to achieve business outcomes, not just the model weights.

## Topics

Agentic AI, Reinforcement Learning (RL), Model Post-Training, MLOps, System Architecture, Benchmark Design, Hugging Face Slides, Training Agents IRL Intro, Learnings from enterprise RL, How to train your agents with Miles, Why we built OpenEnv Arena, Full curriculum (blog)

Source: https://www.youtube.com/watch?v=CKFUx7COIwQ