Scaling Compute on Context — Jack Morris, Engram
Summary
This talk introduces the concept of 'Scaling Compute on Context,' addressing the fundamental limitation that current Large Language Models (LLMs) are trained exclusively on public data. While traditional deep learning progress relies on scaling three axes—more data, more compute, and bigger models—the inability to scale with private corporate or personal data limits model depth. The speaker reviews several advanced techniques (e.g., KV compaction, on-policy distillation, continued pretraining) aimed at transferring domain expertise from a fixed, unstructured corpus ($D$) into the model's weights, ultimately seeking a self-improving training mechanism analogous to AlphaGo.
Key takeaways
-
The Knowledge Gap: Public vs. Private Data
5:59
Current LLMs are superb on public data (Wikipedia, GitHub) but lack deep knowledge of private corporate or personal contexts (emails, internal meetings). This is the core problem that limits model depth and expertise acquisition.
-
Scaling Compute on Context
8:40
Since scaling with new public data is limited, the focus shifts to maximizing compute power given a fixed, private dataset ($D$). This approach aims to imbue the pre-trained model ($ heta$) with deep domain knowledge from $D$.
-
The Goal: Self-Improvement and Depth Scaling
11:45
Most current methods eventually saturate (hitting a 'data wall'). The desired property is self-improvement, where getting better makes the training questions harder, allowing compute to continuously buy depth rather than flattening out.
Technical details
-
Scaling Axes Limitation
520s
The three traditional axes of scaling (more data, more compute, bigger models) are limited when dealing with private data because the data source is fixed. This leaves 'compute' as the primary axis for progress on internal corpora.
-
Knowledge Transfer Techniques
615s
Several approaches attempt to simulate knowledge acquisition from context: KV compaction (compressing long contexts), On-policy distillation (making the model think raw text is in context), and Continued pretraining on synthetic data conditioned on $D$.
-
The Saturation Problem
820s
Training a model solely on a fixed dataset ($D$) eventually leads to saturation. The goal is to find a mechanism for self-improvement or recursive training that prevents the model from simply memorizing the data and losing generalization properties.
Mentioned resources
- Ngram
- Cartridges paper (Self-study)
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.