# Stanford EE292R AI and Data for Science | Autumn 2026 | From World Models to World Agents

## Executive summary

The talk reviews the frontier of Physical AI, focusing on World Models—real-time, action-conditioned, causal video generators. These models represent a shift from traditional, explicit 3D simulators to data-driven generative approaches. Key technical concepts include using diffusion models (trained on massive datasets like LAION-5B) to generate photorealistic video conditioned on text, past frames (context), and actions. Major challenges discussed are managing long context windows (due to quadratic complexity) and mitigating 'drift' or 'exposure bias' that occurs when the model generates its own past frames. Solutions involve techniques like Backwards Aggregation (Bagger) and anchoring the model to persistent 3D maps for improved consistency.

## Key takeaways

- World Models vs. Traditional Simulators: World models offer photorealistic appearance and realistic dynamics, unlike traditional simulators which have explicit 3D state representations but often suffer from a 'sim-to-real gap' (e.g., inability to simulate complex fluid flows in real time).
- Diffusion Models for Generation: Generative video models are often based on diffusion models, which learn the flow from a simple, easily sampled distribution (Gaussian noise) to the complex data distribution. This process is typically performed in a latent space using a Variational Autoencoder (VAE).
- Addressing Context and Consistency: To maintain consistency over long periods, models must condition on multiple inputs: text prompts, action signals (e.g., arrow keys, tracked hands), and previous frames (context). For long context, the quadratic complexity of the Transformer's self-attention mechanism is a major bottleneck.
- Mitigating Drift (Exposure Bias): A critical problem is 'drift' or 'exposure bias,' where errors accumulate because the model is trained on clean, real data but generates progressively worse, hallucinated past frames during inference. Solutions include Dataset Aggregation (Dagger) or Backwards Aggregation (Bagger) by training on degraded or reversed versions of the model's own output.

## Technical details

- World Model Architecture: A video world model is defined as a real-time, action-conditioned, causal (auto-regressive) video generator. It uses a diffusion model architecture, often implemented with a Transformer, to predict the next frame based on the current state and conditioning signals.
- Diffusion Process Mechanics: The process involves learning the reverse diffusion: starting from noise and iteratively estimating the velocity field (denoiser) to move toward a clean data sample. This is often done in a latent space after passing the pixel image through a Variational Autoencoder (VAE).
- State and Context Management: For long-term consistency, the model must be anchored. Techniques include using a persistent 3D map (e.g., a point cloud) to condition the generation, which helps prevent the model from hallucinating the environment's structure when the agent revisits a location.
- Controllability and Control Signals: Advanced control features include explicit camera control (specifying where the camera looks), hand control (for generated reality/VR), and space-time control (allowing independent control over global time and camera movement, like 'bullet time').

## Practical implications

- **Autonomous Systems:** World models are used to train and evaluate physical AI (robots, self-driving cars) by providing vast, plausible, and diverse simulated data, mitigating the 'sim-to-real gap.'
- **Data Augmentation:** Companies use these models to generate rare or difficult-to-observe events (e.g., an elephant on a street) to make training policies more robust.
- **Industrial Automation:** The technology enables robots to perform complex, multi-step tasks in logistics and manufacturing (e.g., decanting, sorting), moving beyond simple, pre-programmed movements.
- **Virtual/Augmented Reality:** World models can generate entire interactive experiences ('generated reality') conditioned on real-time inputs like tracked hands and head pose.

## Topics

Generative AI, World Models, Diffusion Models, Physical AI, Robotics, Long Context Modeling, Auto-regressive Generation, Genie 3, Waymo / Tesla, Nvidia Cosmos, LAION-5B, Oasis, Marlo

Source: https://www.youtube.com/watch?v=IfkeGU4bEJI