# Scaling to Long Horizons — Ross Taylor & Chengxi Taylor, General Reasoning

## Executive summary

The talk discusses scaling AI agents to solve long-horizon problems, arguing that success requires a shift in mindset from merely increasing context windows to improving environment quality and algorithmic efficiency. Key technical advancements include using value models (critics) to reduce gradient variance and employing techniques like compaction and bootstrapping to manage sparse rewards over extended trajectories. The speakers emphasize that real-world complexity and robust simulation environments are more critical than simply having larger base models.

## Key takeaways

- Long Horizon is a Mindset, Not Just an Engineering Problem: Solving major human challenges (e.g., curing cancer) requires patience and thinking in long timeframes, necessitating a fundamental shift in how AI systems are designed for sustained coherence.
- RLHF is Crucial for Productizing LLMs: The breakthrough that made LLMs usable was not the base model size, but the application of Reinforcement Learning from Human Feedback (RLHF), which provided necessary alignment and structure.
- Value Models Mitigate Long-Horizon Challenges: To handle long trajectories, value models (critics) are essential for reducing gradient variance and facilitating credit assignment, which is necessary when rewards are sparse.
- Simulation Quality Trumps Context Window Size: The failure of frontier models in real-world tasks (like trading football matches) demonstrated that scaling requires better, more complex environments and simulations, not just larger context windows.

## Technical details

- Galactica's Contributions: The Galactica model demonstrated key concepts like data efficiency (using smaller, curated corpora) and multi-epoch training, showing that high-quality data is paramount.
- Thinking Tokens/Internal Memory: The concept of 'thinking tokens' suggests an internal working memory process (e.g., using specific tags) that allows the agent to perform explicit, structured reasoning before generating a final answer.
- RL Techniques for Long Trajectories: To manage long rollouts and sparse rewards, techniques include: Compaction (summarizing tokens to fit within context limits), Value Models/Critics (reducing variance), Bootstrapping (generating expected signal before the episode ends), and File System Tools (acting as an AI scratchpad).
- Compute Trade-offs in Pipeline RL: Pipeline RL involves generating sequences while simultaneously training the model. This creates off-policy data, but there is a trade-off between maximizing off-policy steps and maintaining optimal GPU utilization, especially when inference takes weeks.

## Practical implications

- For build engineers designing AI systems, the focus must shift from simply increasing context window size to building robust, complex simulation environments that mimic real-world uncertainty and multiple interacting agents.
- System optimization for long-horizon tasks requires careful resource management, balancing off-policy data generation with GPU utilization through techniques like value model bootstrapping.
- The implementation of structured reasoning mechanisms (like 'thinking tokens' or file system tools) should be prioritized to ensure the agent performs deliberate internal thought processes rather than just retrieving previous answers.

## Topics

Reinforcement Learning, Large Language Models (LLMs), AI Scaling, Compute Optimization, Simulation Environments, openreward.ai, General Reasoning (GR)

Source: https://www.youtube.com/watch?v=2bvtay8wGYI