Scaling to Long Horizons — Ross Taylor & Chengxi Taylor, General Reasoning
The talk discusses scaling AI agents to solve long-horizon problems, arguing that success requires a shift in mindset from merely increasing context windows to improving environment quality and algorithmic efficiency. Key technical advancements include using value models (critics) to reduce gradient variance and employing techniques like compaction and bootstrapping to manage sparse rewards over extended trajectories. The speakers emphasize that real-world complexity and robust simulation environments are more critical than simply having larger base models.
Key takeaways
-
Long Horizon is a Mindset, Not Just an Engineering Problem
15:12
Solving major human challenges (e.g., curing cancer) requires patience and thinking in long timeframes, necessitating a fundamental shift in how AI systems are designed for sustained coherence.
-
RLHF is Crucial for Productizing LLMs
4:00
The breakthrough that made LLMs usable was not the base model size, but the application of Reinforcement Learning from Human Feedback (RLHF), which provided necessary alignment and structure.
-
Value Models Mitigate Long-Horizon Challenges
16:56
To handle long trajectories, value models (critics) are essential for reducing gradient variance and facilitating credit assignment, which is necessary when rewards are sparse.
-
Simulation Quality Trumps Context Window Size
The failure of frontier models in real-world tasks (like trading football matches) demonstrated that scaling requires better, more complex environments and simulations, not just larger context windows.