# Stanford CS329A Self-Improving AI Agents | Part 4 | Learning from Feedback with Tools/Code

## Executive summary

This lecture details three advanced methodologies for building self-improving AI agents: ReAct, RLEF, and Constitutional AI. These techniques enable Large Language Models (LLMs) to move beyond simple text generation by incorporating external feedback loops—whether through environment interaction, code execution results, or self-critique. For build engineering, the core takeaway is that robust agentic behavior requires structured methods for grounding reasoning in real-world data and iterative refinement using test feedback.

## Key takeaways

- ReAct: Combining Reasoning and Action: The ReAct framework combines Chain-of-Thought (reasoning) with tool-calling actions, creating an interleaved loop (thought $ ightarrow$ act $ ightarrow$ observe). This allows models to ground their reasoning in external knowledge sources, improving performance on tasks like HotpotQA and WebShop. [0:03]
- RLEF: Grounding Code LLMs via Execution Feedback: RLEF is an end-to-end RL fine-tuning framework for coding agents that uses execution feedback (test pass/fail) as a binary reward signal. It employs a two-tier test strategy, using public tests for inference-time guidance and private tests for training the policy model via PPO. [0:12]
- Constitutional AI: Self-Correction through Principles: This method improves LLMs by having them critique their own outputs against a set of human-written principles (the 'Constitution'). This generates feedback for training the model's preference model, reducing reliance on constant human labeling. [0:15]

## Technical details

- ReAct Architecture: The ReAct abstraction facilitates a loop where reasoning traces are generated (left loop) and actions are taken based on those thoughts (right loop). This is crucial for tasks requiring real-world grounding, such as fact-checking (FEVER) or web interaction (WebShop). [0:03]
- RLEF Training Loop: The framework uses an iterative feedback cycle where the model generates code $ ightarrow$ public tests provide immediate execution feedback $ ightarrow$ the LLM attempts to fix the solution $ ightarrow$ a private test set determines the final reward for PPO training. This separation prevents overfitting to public test outputs. [0:12]
- Constitutional AI Process: The process involves three stages: (1) Supervised fine-tuning on self-critique/revision traces; (2) Training a preference model using the Constitution principles; and (3) Fine-tuning the LLM to maximize adherence to these principles, achieving better helpfulness/harmlessness trade-offs. [0:15]

## Practical implications

- The concepts of ReAct and RLEF provide blueprints for building robust, multi-step agents capable of interacting with external APIs or code environments.
- The use of execution feedback (unit tests) in RL fine-tuning is directly applicable to improving automated testing within CI/CD pipelines.
- Constitutional AI offers a scalable approach to guiding model behavior by defining explicit rulesets rather than relying solely on massive human labeling efforts.

## Topics

Agentic LLMs, Reinforcement Learning from Feedback (RLHF), Tool Calling, Code Generation, Self-Improvement, HotpotQA, FEVER, WebShop, CodeContests

Source: https://www.youtube.com/watch?v=Lxh9RF5S-K0