# Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI

## Executive summary

In an era of increasing compute scarcity—evidenced by rising H100 prices and skyrocketing token usage—data quality has emerged as the critical 'compute multiplier' for model training. The presentation outlines a systematic approach to data enhancement through four stages: Clean, Curate, Create, and Compose. By maximizing the signal per token (marginal information gain), organizations can achieve performance levels comparable to models trained with vastly more compute budgets. Practical applications include improving Vision Language Models (VLMs) and enhancing multilingual capabilities using proprietary or public datasets.

## Key takeaways

- Compute Scarcity Drives Data Focus: The availability of compute is becoming increasingly constrained, leading to market actions like Google capping Meta's Gemini usage and OpenAI selling token futures. This necessitates a shift in focus from raw compute power to data quality.
- Data Quality as Compute Multiplier: Improving data quality allows for dramatically better performance (blue curve) compared to training with the same limited compute budget (gray curve), effectively simulating much larger compute investments.
- The Four C's of Data Enhancement: Data improvement is achieved through a pipeline: Clean (heuristic filters, decontamination), Curate (quality classifiers, redundancy reduction), Create (synthetic data generation/rephrasing), and Compose (sequencing across multiple training stages).
- Cross-Lingual Benefits from Curation: Curating English data can positively benefit non-English performance, demonstrating cross-lingual transfer. Similarly, curating non-English data benefits English performance.

## Technical details

- Data Optimization Goal: The fundamental goal is to maximize the marginal information gain per data point shown to the model, ensuring relevance to specific target use cases.
- VLM Performance Gains: Curating a dataset (e.g., using the Mammoth dataset) can yield an absolute 14 percentage point improvement in VLM performance while requiring significantly less training compute compared to public benchmarks like Quen 3.5b.
- Multilingual Scaling: Curating data is critical for non-Western use cases, allowing models to achieve strong multilingual MMLU performance using only a small fraction of the total tokens (e.g., 8% of data used as multilingual tokens).
- Synthetic Data Generation: The technique of 'rephrasing' converts proprietary documents into multiple formats (e.g., true/false questions), increasing diversity and allowing training on high-quality, structured data points without model collapse.

## Practical implications

- Organizations should prioritize investment in data curation and quality over simply increasing compute budgets to achieve better model performance.
- The training pipeline must be viewed synergistically (pre-training $ ightarrow$ mid-training $ ightarrow$ post-training) rather than as independent stages.
- High-quality, proprietary data can significantly boost domain-specific capabilities (e.g., legal reasoning) and make subsequent fine-tuning steps much more effective.

## Topics

AI Model Training, Data Engineering, Large Language Models (LLMs), Compute Optimization, DatologyAI, H100, Gemini, Meta's Gemini usage cap, Quen 3.5b, Thomson Reuters, RCI Trendy Large

Source: https://www.youtube.com/watch?v=_PdK6x7PQNM