Stanford CS329A Self-Improving AI Agents | Part 4 | Learning from Feedback with Tools/Code
This lecture details three advanced methodologies for building self-improving AI agents: ReAct, RLEF, and Constitutional AI. These techniques enable Large Language Models (LLMs) to move beyond simple text generation by incorporating external feedback loops—whether through environment interaction, code execution results, or self-critique. For build engineering, the core takeaway is that robust agentic behavior requires structured methods for grounding reasoning in real-world data and iterative refinement using test feedback.
Key takeaways
-
ReAct: Combining Reasoning and Action
0:03
The ReAct framework combines Chain-of-Thought (reasoning) with tool-calling actions, creating an interleaved loop (thought $ ightarrow$ act $ ightarrow$ observe). This allows models to ground their reasoning in external knowledge sources, improving performance on tasks like HotpotQA and WebShop. [0:03]
-
RLEF: Grounding Code LLMs via Execution Feedback
0:12
RLEF is an end-to-end RL fine-tuning framework for coding agents that uses execution feedback (test pass/fail) as a binary reward signal. It employs a two-tier test strategy, using public tests for inference-time guidance and private tests for training the policy model via PPO. [0:12]
-
Constitutional AI: Self-Correction through Principles
0:15
This method improves LLMs by having them critique their own outputs against a set of human-written principles (the 'Constitution'). This generates feedback for training the model's preference model, reducing reliance on constant human labeling. [0:15]