Topic

Transformer Models

All digests tagged Transformer Models

OpenAI Astra and Recurrent Depth / Looped Transformers

This video explains looped transformers—a technique to scale LLM depth by reusing weights multiple times. We analyze fixed repetition (Nanbeige4.2-3B) vs. dynamic Mixture-of-Recursions.

Read article

Umar Jamil

Building a distributed training framework from first principles

This deep dive builds a distributed training framework from first principles using PyTorch. Topics include RoPE, YaRN, MoE, and combining multiple forms of parallelism to train modern LLMs.

Read article