# Why Nobody Has Solved Continual Learning

## Executive summary

The core challenge in 'owning your intelligence' is the continual learning loop, which the speaker asserts remains unsolved. The process requires balancing Supervised Fine-Tuning (SFT), Reinforcement Learning (RL), and distillation. SFT is excellent for installing new knowledge but is highly off-policy, leading to catastrophic forgetting when deployed in production edge cases. Conversely, RL is highly on-policy and stable but is limited to upweighting behaviors the model can already sample, making it difficult to bake in novel knowledge. The optimal strategy involves carefully tuning the ratio of SFT/Distillation to RL for the specific task.

## Key takeaways

- The Continual Learning Loop is Unsolved: The 'train box' for continually updating model weights (the continual learning regime) is extremely difficult. No single method (SFT or RL) provides a perfect solution for continuous improvement.
- SFT is Off-Policy and Fragile: SFT allows installing new knowledge (e.g., using gold traces from Fable 5.1 on Qwen 4B), but because it is off-policy, the model fails catastrophically when encountering out-of-distribution (OOD) production edge cases, as it has not been trained on its own mistakes.
- RL is On-Policy but Limited in Scope: RL is highly on-policy, meaning it only updates based on samples generated by the current model, which prevents catastrophic forgetting. However, it cannot bake in novel knowledge if the model is unlikely to sample a correct rollout in the first place.
- Distillation as a Warm Start: Distillation is used to warm-start a model by incorporating reasoning traces from a closed-source model. This provides a non-zero probability of sampling a good rollout, allowing RL to begin its climb. However, the speaker notes that RLing on a distilled model plateaus lower than the model that was RL'd without distillation.
- The Importance of the SFT/Distillation to RL Ratio: The most critical takeaway is that the performance of the model depends on being 'very smart about the ratio of SFT/distillation to RL' for the specific task at hand.

## Technical details

- Supervised Fine-Tuning (SFT): SFT uses 'gold traces' (e.g., from Fable 5.1) to update a model (e.g., Qwen3 4B). It is effective for installing new information but is inherently off-policy, making it susceptible to failure when encountering OOD data.
- Reinforcement Learning (RL): RL updates are constrained to the model's current support (on-policy), which prevents catastrophic forgetting. The mechanism involves upweighting positive rollouts and downweighting negative ones. This stability is due to an implicit KL penalty tying the model closely to its base weights.
- Catastrophic Forgetting: This is a major failure mode of continual SFT, where the model forgets previously learned knowledge (both general pre-training knowledge and task-specific knowledge) when fine-tuned on new, distinct tasks.
- Self-Distillation: A technique used to provide 'hints' or dense-level information to the model during RL, attempting to combine the benefits of both SFT (knowledge injection) and RL (optimization). However, it can introduce bias into the RL algorithm.

## Practical implications

- For narrow, stationary tasks (e.g., clinical note generation), repeated SFT on generated 'gold traces' may be effective.
- For general, agentic tasks (e.g., universal Turing LLM machine), the model must undergo post-training on multiple environments and potentially mid-training updates, requiring a frontier lab approach.
- The primary engineering focus should be on determining the optimal ratio of SFT/Distillation to RL for the specific application, rather than relying on a single training paradigm.

## Topics

Continual Learning, Large Language Models (LLMs), Model Training, Reinforcement Learning (RL), Supervised Fine-Tuning (SFT), Model Deployment, AI Architecture, Baseten, DAgger paper, Cartridges paper (self-study)

Source: https://www.youtube.com/watch?v=whnDiwcL4HE