Compute Optimization

Jul 31, 2026 · 19:05

Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI

In an era of increasing compute scarcity—evidenced by rising H100 prices and skyrocketing token usage—data quality has emerged as the critical 'compute multiplier' for model training. The presentation outlines a systematic approach to data enhancement through four stages: Clean, Curate, Create, and Compose. By maximizing the signal per token (marginal information gain), organizations can achieve performance levels comparable to models trained with vastly more compute budgets. Practical applications include improving Vision Language Models (VLMs) and enhancing multilingual capabilities using proprietary or public datasets.

Key takeaways

1. Compute Scarcity Drives Data Focus

   The availability of compute is becoming increasingly constrained, leading to market actions like Google capping Meta's Gemini usage and OpenAI selling token futures. This necessitates a shift in focus from raw compute power to data quality.

2. Data Quality as Compute Multiplier 3:39

   Improving data quality allows for dramatically better performance (blue curve) compared to training with the same limited compute budget (gray curve), effectively simulating much larger compute investments.

3. The Four C's of Data Enhancement 5:48

   Data improvement is achieved through a pipeline: Clean (heuristic filters, decontamination), Curate (quality classifiers, redundancy reduction), Create (synthetic data generation/rephrasing), and Compose (sequencing across multiple training stages).

4. Cross-Lingual Benefits from Curation 15:24

   Curating English data can positively benefit non-English performance, demonstrating cross-lingual transfer. Similarly, curating non-English data benefits English performance.

• AI Engineer
• AI Model Training
• Data Engineering
• Large Language Models (LLMs)
• Compute Optimization
• DatologyAI
• H100
• Gemini
• Meta's Gemini usage cap
• Quen 3.5b
• Thomson Reuters
• RCI Trendy Large

Watch on YouTube Full article

Jul 31, 2026 · 18:07

Scaling to Long Horizons — Ross Taylor & Chengxi Taylor, General Reasoning

The talk discusses scaling AI agents to solve long-horizon problems, arguing that success requires a shift in mindset from merely increasing context windows to improving environment quality and algorithmic efficiency. Key technical advancements include using value models (critics) to reduce gradient variance and employing techniques like compaction and bootstrapping to manage sparse rewards over extended trajectories. The speakers emphasize that real-world complexity and robust simulation environments are more critical than simply having larger base models.

Key takeaways

1. Long Horizon is a Mindset, Not Just an Engineering Problem 15:12

   Solving major human challenges (e.g., curing cancer) requires patience and thinking in long timeframes, necessitating a fundamental shift in how AI systems are designed for sustained coherence.

2. RLHF is Crucial for Productizing LLMs 4:00

   The breakthrough that made LLMs usable was not the base model size, but the application of Reinforcement Learning from Human Feedback (RLHF), which provided necessary alignment and structure.

3. Value Models Mitigate Long-Horizon Challenges 16:56

   To handle long trajectories, value models (critics) are essential for reducing gradient variance and facilitating credit assignment, which is necessary when rewards are sparse.

4. Simulation Quality Trumps Context Window Size

   The failure of frontier models in real-world tasks (like trading football matches) demonstrated that scaling requires better, more complex environments and simulations, not just larger context windows.

• AI Engineer
• Reinforcement Learning
• Large Language Models (LLMs)
• AI Scaling
• Compute Optimization
• Simulation Environments
• openreward.ai
• General Reasoning (GR)

Watch on YouTube Full article