# The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside

## Executive summary

The presentation details how poolside scaled their LLM training from smaller models (Laguna M/XS) to a massive 118 billion parameter model for agentic coding. The core strategy involves creating sophisticated synthetic data pipelines—which complement organic data by exposing implicit rationale and structure—and implementing extreme rigor in the pre-training process. Key technical challenges addressed include catching silent failures like broken GPUs, numerical precision loss during tensor parallel accumulation (BF16 to FP32 fixes), and race conditions via weight hashing checks. The resulting model, Laguna S, demonstrates competitive performance against industry leaders on coding benchmarks.

## Key takeaways

- Synthetic Data for Feature Extraction: Synthetic data is used not to replace organic data but to complement it by extracting and projecting implicitly hidden features (e.g., implicit rationale, planning, structure). Pipelines are modular, allowing complex tasks to be broken down into stages like multi-stage workflows, cross-domain porting (e.g., math problems to code), or multi-turn chats.
- Rigorous Training Validation: Trust Nothing: To ensure correctness at scale, the team implemented model replica hashing checks; if hashes over weights do not match across distributed replicas, training is immediately halted. Failures observed include silent data corruption from broken GPUs and gradient corruption due to race conditions.
- Scaling Success: The 118B Parameter Model: The final model, Laguna S (118B parameters), was trained on 30 trillion tokens across 4,000 GPUs. Early results show strong performance in agentic coding benchmarks (e.g., SWEETbench), outperforming models like GLM 4.5 Air.

## Technical details

- Data Pipeline Architecture: Synthetic pipelines are modular, composed of six components: seeds (primary inputs), metadata, secondary inputs, generator/function (which can be an agent with tools or prompt templates), and supplementary functions (filters/validators). Pipelines range from cheap, seed-heavy rephrasing to complex workflows requiring orchestration.
- Multi-Stage Workflows: Complex tasks are broken down into sequential steps (e.g., generating a novel by first setting the plot/characters, then writing chapters). This allows for better generation quality and controlled aggregation of processing.
- Numerical Stability in Training: A critical failure point was observed when using tensor parallel accumulation in BF16, where growing activations led to insufficient numerical precision. The fix involved moving the accumulation process into FP32, allowing convergence to resume.
- Race Condition Detection: A silent corruption of approximately 0.5% of the gradient was traced back to issues in added FP8 training kernels (DeepChem). This highlights that standard weight hashing checks cannot detect race conditions, necessitating new 'blind spot' checking mechanisms.

## Practical implications

- For ML infrastructure teams, the necessity of implementing robust, multi-layered validation (beyond simple weight hashing) is critical for scaling LLMs.
- The modular pipeline design suggests that complex data generation and model training can be broken down into configurable, orchestrated microservices, improving reproducibility and debugging.
- Build engineers must account for numerical precision issues (e.g., BF16 vs FP32) when designing distributed training pipelines to prevent silent convergence failures.

## Topics

Large Language Models, Synthetic Data Generation, Distributed Training, Model Scaling, Agentic AI, MLOps, Laguna M / Laguna XS, XGen-2, Laguna S (118B parameters)

Source: https://www.youtube.com/watch?v=KhYifX22yhE