Workshop: Training Agents at the frontier of AI Industry
Summary
This workshop provides an in-depth look at the industrial challenges and advanced methodologies for post-training large language models (LLMs) into functional agents. Speakers cover everything from enterprise system design (latency, de-risking, containment) to advanced RL techniques (SFT, DPO, distillation, GRPO). Key focus areas include building robust, measurable environments, scaling training on massive models (up to 320B parameters), and establishing centralized, vetted resources like OpenEnv Arena to combat issues like reward hacking and answer leakage.
Key takeaways
-
System Design is Paramount for Enterprise Agents
For enterprise use cases, success depends less on the model itself and more on the entire system, including tools, data, network, and wall-clock time. Optimization requires traditional distributed systems engineering techniques like parallelizing API calls, introducing caching layers, and using materialized views to improve latency and accuracy. Containment and least privilege are critical security concerns.
-
Distinguishing Rewards from Measurements in Evaluation
32:03
A critical distinction must be made between a 'reward' (used during training to guide the model) and a 'measurement' (used during evaluation to assess reliability). Training environments should be designed to produce verifiable reward signals, while evaluation benchmarks must be rigorously assessed for 'fitness to purpose' to ensure the score predicts real-world performance.
-
Scaling Agent RL with Fully Asynchronous Infrastructure
1:07:05
Training agents requires handling multi-step interactions (trajectories) rather than single responses. Frameworks like Miles support this by enabling 'fully async RL,' where inference and training run on separate GPU groups. This prevents slow episodes from idling the training GPUs, which is crucial when scaling to large models (e.g., GLM 5.3, 320B parameters).
-
The Need for Vetted, Scalable Environments
1:27:20
The complexity of post-training requires centralized, audited resources. OpenEnv Arena aims to vet environments for trainability, checking for failure modes like 'answer leakage' and 'reward hacking.' The process involves filtering vast numbers of potential environments to ensure they provide a reliable, non-trivial training signal.
Technical details
-
Post-Training Techniques
1923s
Methods include Supervised Fine-Tuning (SFT) using good traces, Direct Preference Optimization (DPO) using pairwise preferences (e.g., thumbs up/down data), and distillation (e.g., reverse KL distillation). Reinforcement Learning (RL) uses policy gradient methods like GRPO, which must be carefully implemented to prevent divergence.
-
Agentic System Architecture
5240s
Modern agentic systems are highly modular, often relying on conditional logic and routing between specialized, smaller decision models (e.g., using a small LM3 model for routing, a small vision model for vision decisions, and a separate reasoning model for planning). This approach is preferred over using a single large model for all tasks.
-
MLOps and Development Practices
The industry is shifting from traditional ML development to MLOps, requiring robust CI/CD practices for statistical models. This includes maintaining comprehensive regression suites, implementing must-pass tests (for conformity), and adopting a modular, 'pod' structure that integrates Domain Experts (FDEs), ML Engineers, and IT Security teams.
-
Training Data Integrity
1923s
When building benchmarks, it is crucial to ensure the task is 'gradable' and has clear right/wrong outcomes. The process must account for potential data flaws, such as 'answer leakage' (where the answer is present in the environment's source code/history) or 'false positives/negatives' in the reward signal.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.