The Video Signal

Topic

Distributed Training

All digests tagged Distributed Training

Building a distributed training framework from first principles thumbnail

· 19:34:45

Building a distributed training framework from first principles

This video provides a comprehensive, first-principles derivation and implementation guide for building modern distributed training frameworks using PyTorch. The content covers advanced topics necessary for scaling large language models (LLMs), including various forms of parallelism (Data, Tensor, Pipeline, Expert), specialized attention mechanisms (MLA, RoPE), and context window extension techniques (YaRN). A significant portion is dedicated to the mathematical foundations, such as FLOPs calculation ($6NP$) and weight initialization theory, which are crucial for build-engineering teams designing high-performance AI infrastructure.

Key takeaways

  1. Distributed Parallelism Techniques 3:00:00

    The framework combines multiple parallelism types—Pipeline, Data, Tensor, Context, and Expert—into a single working system. This approach ensures that the movement of tensors, communication operations, and gradients are tracked holistically across devices.

  2. Mixture-of-Experts (MoE) 1:43:20

    To scale models without increasing inference latency linearly with parameter count, MoE uses a router to selectively activate only the top-$k$ experts for each token. This concept is vital for achieving compute optimality.

  3. Rotary Position Embeddings (RoPE) 4:10:00

    RoPE encodes positional information by rotating query and key vectors in a complex plane, ensuring that the resulting dot product depends only on the relative distance between tokens, solving the issue of absolute position bias inherent in traditional methods.

  4. YaRN for Context Extension 6:23:20

    To extend context windows (e.g., from 4k to 16k), YaRN addresses RoPE's weakness—the model learning absolute position bias. It interpolates the frequencies, scaling low-frequency dimensions while leaving high-frequency dimensions untouched.

  5. Computational Complexity (FLOPs) 2:50:00

    The FLOPs required for training a transformer model are approximated by $6NP$ (where N is tokens and P is parameters), derived from analyzing the forward and backward passes of matrix multiplication. This metric guides resource allocation and scaling law analysis.

Watch on YouTube Full article

The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside thumbnail

· 17:31

The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside

The presentation details how poolside scaled their LLM training from smaller models (Laguna M/XS) to a massive 118 billion parameter model for agentic coding. The core strategy involves creating sophisticated synthetic data pipelines—which complement organic data by exposing implicit rationale and structure—and implementing extreme rigor in the pre-training process. Key technical challenges addressed include catching silent failures like broken GPUs, numerical precision loss during tensor parallel accumulation (BF16 to FP32 fixes), and race conditions via weight hashing checks. The resulting model, Laguna S, demonstrates competitive performance against industry leaders on coding benchmarks.

Key takeaways

  1. Synthetic Data for Feature Extraction 10:37

    Synthetic data is used not to replace organic data but to complement it by extracting and projecting implicitly hidden features (e.g., implicit rationale, planning, structure). Pipelines are modular, allowing complex tasks to be broken down into stages like multi-stage workflows, cross-domain porting (e.g., math problems to code), or multi-turn chats.

  2. Rigorous Training Validation: Trust Nothing 15:03

    To ensure correctness at scale, the team implemented model replica hashing checks; if hashes over weights do not match across distributed replicas, training is immediately halted. Failures observed include silent data corruption from broken GPUs and gradient corruption due to race conditions.

  3. Scaling Success: The 118B Parameter Model

    The final model, Laguna S (118B parameters), was trained on 30 trillion tokens across 4,000 GPUs. Early results show strong performance in agentic coding benchmarks (e.g., SWEETbench), outperforming models like GLM 4.5 Air.

Watch on YouTube Full article