Model Scaling

Sep 2, 2026 · 28:04

OpenAI Astra and Recurrent Depth / Looped Transformers

This video provides a detailed architectural analysis of the 'recurrent depth' or 'looped transformer' concept, rumored to be used in OpenAI's Astra model. The technique allows for scaling LLM capacity and depth by reusing the same set of weights across multiple passes (loops) through the transformer stack, rather than duplicating all parameters. This approach is compared against traditional fixed-depth models and dynamic methods like Mixture-of-Recursions.

Key takeaways

1. Looped Transformers for Scaling Depth

   The core idea is to increase model depth (e.g., from 22 layers to 44) by reusing the same set of weights multiple times, significantly reducing parameter count compared to duplicating all layers.

2. Nanbeige4.2-3B Implementation 5:02

   This model demonstrates a fixed repetition: passing the input through the same 22-layer stack twice, achieving increased computational depth without doubling the weights.

3. Mixture-of-Recursions (MoR) 20:00

   A dynamic approach where different tokens within a sequence can pass through a variable number of passes (loops), determined at runtime, offering flexibility beyond fixed repetition.

4. Training vs. Retrofitting 12:23

   The speaker asserts that training the model from scratch with the looped transformer aspect is significantly better than retrofitting it onto an already trained architecture, as LLM performance heavily relies on initial training data.

• Sebastian Raschka
• LLM Architecture
• Transformer Models
• Recurrent Neural Networks
• Model Scaling
• Attention Mechanisms
• OpenAI Astra article
• Looped depth sharing explainer
• Nanbeige4.2-3B technical report
• Mixture-of-Recursions paper

Watch on YouTube Full article

Jul 26, 2026 · 17:31

The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside

The presentation details how poolside scaled their LLM training from smaller models (Laguna M/XS) to a massive 118 billion parameter model for agentic coding. The core strategy involves creating sophisticated synthetic data pipelines—which complement organic data by exposing implicit rationale and structure—and implementing extreme rigor in the pre-training process. Key technical challenges addressed include catching silent failures like broken GPUs, numerical precision loss during tensor parallel accumulation (BF16 to FP32 fixes), and race conditions via weight hashing checks. The resulting model, Laguna S, demonstrates competitive performance against industry leaders on coding benchmarks.

Key takeaways

1. Synthetic Data for Feature Extraction 10:37

   Synthetic data is used not to replace organic data but to complement it by extracting and projecting implicitly hidden features (e.g., implicit rationale, planning, structure). Pipelines are modular, allowing complex tasks to be broken down into stages like multi-stage workflows, cross-domain porting (e.g., math problems to code), or multi-turn chats.

2. Rigorous Training Validation: Trust Nothing 15:03

   To ensure correctness at scale, the team implemented model replica hashing checks; if hashes over weights do not match across distributed replicas, training is immediately halted. Failures observed include silent data corruption from broken GPUs and gradient corruption due to race conditions.

3. Scaling Success: The 118B Parameter Model

   The final model, Laguna S (118B parameters), was trained on 30 trillion tokens across 4,000 GPUs. Early results show strong performance in agentic coding benchmarks (e.g., SWEETbench), outperforming models like GLM 4.5 Air.

• AI Engineer
• Large Language Models
• Synthetic Data Generation
• Distributed Training
• Model Scaling
• Agentic AI
• MLOps
• Laguna M / Laguna XS
• XGen-2
• Laguna S (118B parameters)

Watch on YouTube Full article