Building a distributed training framework from first principles
This deep dive builds a distributed training framework from first principles using PyTorch. Topics include RoPE, YaRN, MoE, and combining multiple forms of parallelism to train modern LLMs.
Read article