Scaling to Long Horizons — Ross Taylor & Chengxi Taylor, General Reasoning
Summary
The talk discusses scaling AI agents to solve long-horizon problems, arguing that success requires a shift in mindset from merely increasing context windows to improving environment quality and algorithmic efficiency. Key technical advancements include using value models (critics) to reduce gradient variance and employing techniques like compaction and bootstrapping to manage sparse rewards over extended trajectories. The speakers emphasize that real-world complexity and robust simulation environments are more critical than simply having larger base models.
Key takeaways
-
Long Horizon is a Mindset, Not Just an Engineering Problem
15:12
Solving major human challenges (e.g., curing cancer) requires patience and thinking in long timeframes, necessitating a fundamental shift in how AI systems are designed for sustained coherence.
-
RLHF is Crucial for Productizing LLMs
4:00
The breakthrough that made LLMs usable was not the base model size, but the application of Reinforcement Learning from Human Feedback (RLHF), which provided necessary alignment and structure.
-
Value Models Mitigate Long-Horizon Challenges
16:56
To handle long trajectories, value models (critics) are essential for reducing gradient variance and facilitating credit assignment, which is necessary when rewards are sparse.
-
Simulation Quality Trumps Context Window Size
The failure of frontier models in real-world tasks (like trading football matches) demonstrated that scaling requires better, more complex environments and simulations, not just larger context windows.
Technical details
-
Galactica's Contributions
370s
The Galactica model demonstrated key concepts like data efficiency (using smaller, curated corpora) and multi-epoch training, showing that high-quality data is paramount.
-
Thinking Tokens/Internal Memory
580s
The concept of 'thinking tokens' suggests an internal working memory process (e.g., using specific tags) that allows the agent to perform explicit, structured reasoning before generating a final answer.
-
RL Techniques for Long Trajectories
912s
To manage long rollouts and sparse rewards, techniques include: Compaction (summarizing tokens to fit within context limits), Value Models/Critics (reducing variance), Bootstrapping (generating expected signal before the episode ends), and File System Tools (acting as an AI scratchpad).
-
Compute Trade-offs in Pipeline RL
Pipeline RL involves generating sequences while simultaneously training the model. This creates off-policy data, but there is a trade-off between maximizing off-policy steps and maintaining optimal GPU utilization, especially when inference takes weeks.
Mentioned resources
- openreward.ai
- General Reasoning (GR)
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.