# Scaling Compute on Context — Jack Morris, Engram

## Executive summary

This talk introduces the concept of 'Scaling Compute on Context,' addressing the fundamental limitation that current Large Language Models (LLMs) are trained exclusively on public data. While traditional deep learning progress relies on scaling three axes—more data, more compute, and bigger models—the inability to scale with private corporate or personal data limits model depth. The speaker reviews several advanced techniques (e.g., KV compaction, on-policy distillation, continued pretraining) aimed at transferring domain expertise from a fixed, unstructured corpus ($D$) into the model's weights, ultimately seeking a self-improving training mechanism analogous to AlphaGo.

## Key takeaways

- The Knowledge Gap: Public vs. Private Data: Current LLMs are superb on public data (Wikipedia, GitHub) but lack deep knowledge of private corporate or personal contexts (emails, internal meetings). This is the core problem that limits model depth and expertise acquisition.
- Scaling Compute on Context: Since scaling with new public data is limited, the focus shifts to maximizing compute power given a fixed, private dataset ($D$). This approach aims to imbue the pre-trained model ($ heta$) with deep domain knowledge from $D$.
- The Goal: Self-Improvement and Depth Scaling: Most current methods eventually saturate (hitting a 'data wall'). The desired property is self-improvement, where getting better makes the training questions harder, allowing compute to continuously buy depth rather than flattening out.

## Technical details

- Scaling Axes Limitation: The three traditional axes of scaling (more data, more compute, bigger models) are limited when dealing with private data because the data source is fixed. This leaves 'compute' as the primary axis for progress on internal corpora.
- Knowledge Transfer Techniques: Several approaches attempt to simulate knowledge acquisition from context: KV compaction (compressing long contexts), On-policy distillation (making the model think raw text is in context), and Continued pretraining on synthetic data conditioned on $D$.
- The Saturation Problem: Training a model solely on a fixed dataset ($D$) eventually leads to saturation. The goal is to find a mechanism for self-improvement or recursive training that prevents the model from simply memorizing the data and losing generalization properties.

## Practical implications

- Enterprises can use this framework to build specialized LLMs that possess deep domain expertise from internal documents and communications, moving beyond general public knowledge.
- The concept suggests a new MLOps paradigm where model refinement is driven by continuous, compute-intensive interaction with proprietary data.

## Topics

Large Language Models (LLMs), Continual Learning, Deep Learning Scaling, Domain Adaptation, Ngram, Cartridges paper (Self-study)

Source: https://www.youtube.com/watch?v=WiqDvX6isc4