# Stanford CS329A Self-Improving AI Agents | Part 1 | Course Overview

## Executive summary

This course overview details the evolution of Large Language Models (LLMs) from simple predictive models to complex, self-improving agents. Key advancements include understanding scaling laws based on parameters, compute, and data size; developing advanced reasoning capabilities through Chain-of-Thought (CoT); and moving beyond single-turn chat interactions into sophisticated agentic workflows. The focus shifts to building systems that can plan, interact with external tools, self-correct using verifiers, and achieve end-to-end goals in domains like software engineering and data analysis.

## Key takeaways

- LLM Scaling Laws: Model performance improves significantly by scaling up parameters, compute (compute on the x-axis), or dataset size. This exponential growth has been foundational for models like GPT-3 and PaLM.
- Agentic Workflows vs. Chatbots: Modern LLM applications are moving from single-turn chat formats to complex agentic workflows (e.g., Deep Research, Cloud Code). These systems can take a goal, plan steps, interact with environments/tools, and correct their actions until the goal is achieved.
- The Role of Verifiers: In agentic systems, verifiers (like unit tests for code) are crucial feedback mechanisms. They allow models to verify generated outputs and guide self-correction, which is essential in verifiable domains like math and coding.
- Self-Improvement Loop: The most advanced frontier involves combining test time scaling (e.g., repeated sampling) with fine-tuning to generate synthetic, high-quality data. This creates a self-improving loop where the model trains on its own generated solutions.

## Technical details

- Scaling Laws: LLM performance improves via three axes: increasing parameters, increasing compute (test loss decreases), and increasing dataset size. Examples include GPT-2 (1.5B params) to PaLM (540B params).
- Reasoning & CoT: Chain-of-Thought (CoT) is an emergent property where providing the reasoning steps, not just the answer, significantly improves performance on complex tasks. This capability appears strongly in larger models like PaLM and GPT.
- Model Alignment Techniques: Models are refined through multiple steps: (1) Fine-tuning on high-quality data, (2) Instruction Tuning (learning Q&A pairs), and (3) Reinforcement Learning from Human Feedback (RLHF). RLHF uses a reward model derived from human ratings to guide the LLM's parameters.
- Inference Scaling: This technique, demonstrated by 'Large Language Monkeys,' improves performance without changing model parameters. It involves repeatedly sampling (parallel generation) the model's output and using a verifier/selection mechanism to select the correct answer.
- Agentic Components: Advanced workflows utilize components like Prompt Chaining (decomposing tasks), Routing (directing complex tasks to specialized calls), Parallelization, and Orchestrators (central planning LLMs). The system requires an LLM as a Judge/Critic for evaluation.

## Practical implications

- Software Engineers can use coding agents (like Cloud Code) to automate complex tasks such as implementing tests or modifying files based solely on natural language instructions.
- The concept of 'LLM as a Judge' allows for automated evaluation and self-correction in workflows where human feedback is impractical, mimicking unit testing processes.
- Agents can be deployed for repetitive enterprise tasks like code migrations, data warehouse migration, and comprehensive literature reviews, significantly reducing manual effort.

## Topics

Large Language Models (LLMs), Scaling Laws, Chain-of-Thought Reasoning, Reinforcement Learning from Human Feedback (RLHF), Agentic Systems, Workflow Orchestration, Verifiers and Self-Correction, cs329a.stanford.edu, XCS329 graduate course, Large Language Monkeys project

Source: https://www.youtube.com/watch?v=6YnLB0XbTnI