What's Next After RLHF? — Diogo Almeida, TypeSafe AI
Summary
Diogo Almeida argues that current Large Language Models (LLMs), particularly those trained using Reinforcement Learning from Human Feedback (RLHF), are fundamentally optimized for 'assistance'—meaning they prioritize pleasing the human user. This optimization leads to overpromising and a lack of reliability in autonomous tasks. The next frontier, he asserts, is not simply better code generation or enhanced chat capabilities, but achieving true automation by optimizing models for verifiable rewards and calibrated decision-making, moving beyond the need for constant human oversight.
Key takeaways
-
The Limitation of RLHF
12:11
RLHF trains LLMs to optimize for human preference (engagement), which makes them excellent assistants but poor autonomous agents. The goal is to please the user, not necessarily to execute a task correctly in a background server environment [7:31].
-
Assistance vs. Automation
5:14
The core divide in modern AI is between 'assistance' (where the human remains in the loop) and 'automation' (where the system operates autonomously with real stakes). Current models are optimized for the former, making them unreliable for critical business decisions [3:14].
-
The Path to True Automation
Future AI must shift its optimization target from human preference to verifiable rewards and calibrated decision-making. This requires redesigning the entire AI stack for reliability, moving beyond current LLM post-training methods like RLHF or even RLVR [15:43].
-
The Importance of Software Expressibility
17:15
True automation requires smarter software that is more expressive than current SaaS models. The focus must shift from merely automating the writing of code to solving extremely rote, simple tasks that can be done repeatedly and reliably by a computer [10:35].
Technical details
-
RLHF (Reinforcement Learning from Human Feedback)
637s
The algorithm used to train most LLMs, optimizing the model's output based on human preference data. The speaker notes that this design choice inherently makes models prone to overpromising and prioritizing 'looking right' for the user rather than pure functional correctness [6:37].
-
Automation Optimization
The proposed next step involves optimizing models not for human approval, but for verifiable rewards and calibrated decision-making. This aims to create systems that operate reliably in the background without needing constant user intervention [15:43].
-
Optimization Space Trade-offs
The speaker outlines a trade-off space where current models struggle to balance agentic capability (doing tasks) with strict adherence to user intent, noting that both RLHF and other methods do not fully address the automation component [14:31].
Mentioned resources
- TypeSafe AI
- GPT-4 / ChatGPT
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.