Scaling Compute on Context — Jack Morris, Engram
This talk introduces the concept of 'Scaling Compute on Context,' addressing the fundamental limitation that current Large Language Models (LLMs) are trained exclusively on public data. While traditional deep learning progress relies on scaling three axes—more data, more compute, and bigger models—the inability to scale with private corporate or personal data limits model depth. The speaker reviews several advanced techniques (e.g., KV compaction, on-policy distillation, continued pretraining) aimed at transferring domain expertise from a fixed, unstructured corpus ($D$) into the model's weights, ultimately seeking a self-improving training mechanism analogous to AlphaGo.
Key takeaways
-
The Knowledge Gap: Public vs. Private Data
5:59
Current LLMs are superb on public data (Wikipedia, GitHub) but lack deep knowledge of private corporate or personal contexts (emails, internal meetings). This is the core problem that limits model depth and expertise acquisition.
-
Scaling Compute on Context
8:40
Since scaling with new public data is limited, the focus shifts to maximizing compute power given a fixed, private dataset ($D$). This approach aims to imbue the pre-trained model ($ heta$) with deep domain knowledge from $D$.
-
The Goal: Self-Improvement and Depth Scaling
11:45
Most current methods eventually saturate (hitting a 'data wall'). The desired property is self-improvement, where getting better makes the training questions harder, allowing compute to continuously buy depth rather than flattening out.