Deep Learning Optimization

Umar Jamil

Building a distributed training framework from first principles

This deep dive builds a distributed training framework from first principles using PyTorch. Topics include RoPE, YaRN, MoE, and combining multiple forms of parallelism to train modern LLMs.