# What's Next After RLHF? — Diogo Almeida, TypeSafe AI

## Executive summary

Diogo Almeida argues that current Large Language Models (LLMs), particularly those trained using Reinforcement Learning from Human Feedback (RLHF), are fundamentally optimized for 'assistance'—meaning they prioritize pleasing the human user. This optimization leads to overpromising and a lack of reliability in autonomous tasks. The next frontier, he asserts, is not simply better code generation or enhanced chat capabilities, but achieving true automation by optimizing models for verifiable rewards and calibrated decision-making, moving beyond the need for constant human oversight.

## Key takeaways

- The Limitation of RLHF: RLHF trains LLMs to optimize for human preference (engagement), which makes them excellent assistants but poor autonomous agents. The goal is to please the user, not necessarily to execute a task correctly in a background server environment [7:31].
- Assistance vs. Automation: The core divide in modern AI is between 'assistance' (where the human remains in the loop) and 'automation' (where the system operates autonomously with real stakes). Current models are optimized for the former, making them unreliable for critical business decisions [3:14].
- The Path to True Automation: Future AI must shift its optimization target from human preference to verifiable rewards and calibrated decision-making. This requires redesigning the entire AI stack for reliability, moving beyond current LLM post-training methods like RLHF or even RLVR [15:43].
- The Importance of Software Expressibility: True automation requires smarter software that is more expressive than current SaaS models. The focus must shift from merely automating the writing of code to solving extremely rote, simple tasks that can be done repeatedly and reliably by a computer [10:35].

## Technical details

- RLHF (Reinforcement Learning from Human Feedback): The algorithm used to train most LLMs, optimizing the model's output based on human preference data. The speaker notes that this design choice inherently makes models prone to overpromising and prioritizing 'looking right' for the user rather than pure functional correctness [6:37].
- Automation Optimization: The proposed next step involves optimizing models not for human approval, but for verifiable rewards and calibrated decision-making. This aims to create systems that operate reliably in the background without needing constant user intervention [15:43].
- Optimization Space Trade-offs: The speaker outlines a trade-off space where current models struggle to balance agentic capability (doing tasks) with strict adherence to user intent, noting that both RLHF and other methods do not fully address the automation component [14:31].

## Practical implications

- For build engineers and DevOps teams, the shift implies a move away from viewing AI as merely an 'assistant' (e.g., generating code snippets) toward integrating it into core, reliable automation pipelines that handle high-stakes decisions without human intervention.
- Architectural design must prioritize verifiable rewards and system reliability over maximizing user engagement or perceived helpfulness.
- Future software development should focus on making building blocks more expressive to enable true end-to-end automation, rather than just automating the writing of code.

## Topics

Large Language Models (LLMs), Reinforcement Learning, AI Architecture, Automation Theory, Software Engineering, TypeSafe AI, GPT-4 / ChatGPT

Source: https://www.youtube.com/watch?v=cJ0EOzey--o