Why Nobody Has Solved Continual Learning
Summary
The core challenge in 'owning your intelligence' is the continual learning loop, which the speaker asserts remains unsolved. The process requires balancing Supervised Fine-Tuning (SFT), Reinforcement Learning (RL), and distillation. SFT is excellent for installing new knowledge but is highly off-policy, leading to catastrophic forgetting when deployed in production edge cases. Conversely, RL is highly on-policy and stable but is limited to upweighting behaviors the model can already sample, making it difficult to bake in novel knowledge. The optimal strategy involves carefully tuning the ratio of SFT/Distillation to RL for the specific task.
Key takeaways
-
The Continual Learning Loop is Unsolved
The 'train box' for continually updating model weights (the continual learning regime) is extremely difficult. No single method (SFT or RL) provides a perfect solution for continuous improvement.
-
SFT is Off-Policy and Fragile
5:13
SFT allows installing new knowledge (e.g., using gold traces from Fable 5.1 on Qwen 4B), but because it is off-policy, the model fails catastrophically when encountering out-of-distribution (OOD) production edge cases, as it has not been trained on its own mistakes.
-
RL is On-Policy but Limited in Scope
7:10
RL is highly on-policy, meaning it only updates based on samples generated by the current model, which prevents catastrophic forgetting. However, it cannot bake in novel knowledge if the model is unlikely to sample a correct rollout in the first place.
-
Distillation as a Warm Start
10:14
Distillation is used to warm-start a model by incorporating reasoning traces from a closed-source model. This provides a non-zero probability of sampling a good rollout, allowing RL to begin its climb. However, the speaker notes that RLing on a distilled model plateaus lower than the model that was RL'd without distillation.
-
The Importance of the SFT/Distillation to RL Ratio
The most critical takeaway is that the performance of the model depends on being 'very smart about the ratio of SFT/distillation to RL' for the specific task at hand.
Technical details
-
Supervised Fine-Tuning (SFT)
204s
SFT uses 'gold traces' (e.g., from Fable 5.1) to update a model (e.g., Qwen3 4B). It is effective for installing new information but is inherently off-policy, making it susceptible to failure when encountering OOD data.
-
Reinforcement Learning (RL)
430s
RL updates are constrained to the model's current support (on-policy), which prevents catastrophic forgetting. The mechanism involves upweighting positive rollouts and downweighting negative ones. This stability is due to an implicit KL penalty tying the model closely to its base weights.
-
Catastrophic Forgetting
This is a major failure mode of continual SFT, where the model forgets previously learned knowledge (both general pre-training knowledge and task-specific knowledge) when fine-tuned on new, distinct tasks.
-
Self-Distillation
A technique used to provide 'hints' or dense-level information to the model during RL, attempting to combine the benefits of both SFT (knowledge injection) and RL (optimization). However, it can introduce bias into the RL algorithm.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.