OpenAI Astra and Recurrent Depth / Looped Transformers
This video provides a detailed architectural analysis of the 'recurrent depth' or 'looped transformer' concept, rumored to be used in OpenAI's Astra model. The technique allows for scaling LLM capacity and depth by reusing the same set of weights across multiple passes (loops) through the transformer stack, rather than duplicating all parameters. This approach is compared against traditional fixed-depth models and dynamic methods like Mixture-of-Recursions.
Key takeaways
-
Looped Transformers for Scaling Depth
The core idea is to increase model depth (e.g., from 22 layers to 44) by reusing the same set of weights multiple times, significantly reducing parameter count compared to duplicating all layers.
-
Nanbeige4.2-3B Implementation
5:02
This model demonstrates a fixed repetition: passing the input through the same 22-layer stack twice, achieving increased computational depth without doubling the weights.
-
Mixture-of-Recursions (MoR)
20:00
A dynamic approach where different tokens within a sequence can pass through a variable number of passes (loops), determined at runtime, offering flexibility beyond fixed repetition.
-
Training vs. Retrofitting
12:23
The speaker asserts that training the model from scratch with the looped transformer aspect is significantly better than retrofitting it onto an already trained architecture, as LLM performance heavily relies on initial training data.