Foundation Models — The Video Signal

Sep 24, 2026 · 26:42

Robot Demos Are Easy. Reliability Is Hard — Jason Ma, Dyna Robotics

Dyna Robotics focuses on developing highly robust, generalist robotic policies for commercial deployment, arguing that high reliability is more critical than high performance in demos. The company utilizes a 'research and deployment flywheel' to build foundation models, achieving a 99.4% success rate in complex tasks like napkin folding over 24 hours. Key technical advancements include a 'pre-training data pyramid' (over 200,000 hours) and the use of reward models for scalable supervision, allowing the system to detect and recover from errors in long-horizon tasks.

Key takeaways

The Reliability Gap in Robotics 10:10
Achieving a high success rate (e.g., 99.4%) over extended periods (24 hours) is necessary for commercial viability, as standard models often stall at 80–90% success rates, making repeated tasks highly improbable.
The Research and Deployment Flywheel 2:00
Dyna Robotics combines frontier research with active commercial deployments to gather high-quality data, which informs and sharpens the focus of their model development, ensuring the product solves real-world problems.
Scalable Error Recovery via Reward Models 18:59
Instead of relying on manual oversight, the team developed reward models that score the robot's progress during complex tasks. Dips in this score signal a mistake, enabling targeted data collection and a human-in-the-loop active learning cycle for robust error recovery.
Generalization Across Sites
The model architecture is designed to generalize, allowing deployment at new customer sites (e.g., a laundromat, Red Bull events) without requiring site-specific fine-tuning or additional data.

Watch on YouTube Full article

Sep 23, 2026 · 20:42

From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI

The presentation outlines the shift from traditional Vision-Language Models (VLMs) and Vision-Action Models (VALAs) toward Embodied Foundation Models. The core goal is to create unified AI capable of perceiving, reasoning, and acting in the physical world. Key breakthroughs include addressing the massive data inefficiency of long video inputs (token waste) and managing context bloat using data sparse Mixture of Experts (MoE). A major finding is a new scaling law: joint training on perception, reasoning, and control allows a 10x reduction in expensive teleoperation data by increasing video pretraining data.

Key takeaways

Embodied Foundation Models (EFMs)
The north star is building physical AI foundations that unify perception, reasoning, and control into a single model, bridging the physical and digital world. This moves beyond the limitations of separate VLM, VLA, and World Model architectures.
Addressing Data Sparsity and Token Waste 5:40
When modeling long video (e.g., one hour), millions of visual tokens are generated, but ground truth is extremely sparse (e.g., only 2% of tokens are useful). The solution involves developing a 'natural perceptive objective' to automatically predict and focus on percepts that matter (e.g., a gripper tip) rather than treating all background pixels equally.
Context Management via Data Sparse MoE 9:01
To combat context bloat from always-on cameras, the approach utilizes a data sparse Mixture of Experts (MoE) architecture. This includes a router that determines per layer which tokens to read and which to skip, allowing the model to focus compute on task-relevant information.
New Scaling Law for Robotics 9:01
Jointly training on perception, reasoning, and control yields a significant scaling law: 10x more video pretraining data can substitute for 10x less expensive teleoperation (teleop) data, drastically lowering the cost of embodied AI.

Watch on YouTube Full article

Sep 4, 2026 · 27:02

Faster Chips That Don't Melt — Anima Anandkumar & Benedikt Jenik, Accelerated Understanding

Accelerated Understanding proposes a paradigm shift by applying the concept of foundation models—previously successful in language—to the physical world. The core bet is that universality and scale can emerge across diverse physics domains (e.g., fluid dynamics, semiconductors, energy). They are developing single, massive models capable of learning from multiple physical systems simultaneously, achieving unprecedented computational scales like trillion-context training and 5 trillion context inference by utilizing specialized architectures such as neural operators.

Key takeaways

Physical AI Universality
The company aims to create a 'god model' for physics, arguing that common underlying principles (like energy conservation and causality) allow knowledge transfer across vastly different physical domains (e.g., fluid dynamics in catheters vs. nuclear fusion reactors). This shared learning benefits all areas, outperforming models trained on individual domains alone.
Massive Context and Resolution
The model is designed for full 4D rollout (3 spatial dimensions + time). They have achieved the ability to train up to a trillion context input and perform inference at 5 trillion context length, significantly exceeding current capabilities in language or video models.
Self-Improvement via Physics Laws
Unlike purely data-driven AI, the incorporation of physical laws provides a dense, objective training signal. This allows for self-improvement that can push model quality beyond the average quality of the training distribution.

Watch on YouTube Full article

Aug 21, 2026 · 1:11:01

Simulating Humanity: from Generative Agents to 8 Billion Digital Twins — Joon Sung Park, Simile AI

Simile AI aims to simulate human society by creating 'digital twins' of populations, moving beyond current Large Language Model (LLM) capabilities. The core thesis is that predicting human behavior requires modeling underlying 'social physics' and causal mechanisms, not just pattern recognition from web data. The company's approach integrates three primary data types—qualitative interviews, observational/transactional data, and Randomized Controlled Trials (RCTs)—to build highly accurate population and individual-level models. These simulations are intended to help solve 'wicked problems' like climate change and democratic instability by testing policies and interventions before real-world deployment.

Key takeaways

The Ambition: Simulating Society 2:00
The ultimate goal is to simulate the world to answer complex societal questions (e.g., climate change, democratic instability) that are difficult to solve in reality. This is framed as a move from prediction to understanding the path to a desired outcome, similar to Thomas Schelling's work on agent-based modeling.
Modeling Accuracy and Limitations 5:40
Simile claims to have created digital twins that reproduce human behavior and attitudes 85% as accurately as people reproduce their own responses. They argue that frontier LLMs are optimized to be 'super rational,' while human behavior is often irrational, requiring bespoke training on behavioral data.
The Necessity of Causal Data 7:28
To model human decision-making, the most critical data is not just what people say (attitudinal) or what they do (observational), but the data describing the *cause and mechanism* of their decisions, best acquired through Randomized Controlled Trials (RCTs).
Simulation vs. Prediction 10:00
Simulation's highest form is not answering 'what will happen' (prediction), but defining the necessary steps to reach a specific goal (e.g., 'What path must we take to keep unrest to 1,000 years?'). This allows for counterintuitive, yet optimal, interventions.

Watch on YouTube Full article

Jul 31, 2026 · 1:15:57

Stanford CS229 Machine Learning | Spring 2026 | Lecture 12: Representation Learning

The lecture provides a deep dive into advanced generative and adaptation techniques, covering Diffusion Models, Foundation Model paradigms, and Representation Learning. Key technical focus areas include deriving the loss function for diffusion models by comparing true and predicted mean reconstructions ($\mu_{ ilde{t}, t}$ vs $\mu_{ heta, t}$). For large language models (LLMs), the discussion highlights the shift toward using massive unlabelled data for pre-training and adapting via zero-shot or few-shot methods. Finally, it details advanced parameter efficiency techniques like Linear Probing and Low-Rank Adaptation (LoRA), which are crucial for managing memory and compute when deploying large models across multiple users.

Key takeaways

Diffusion Model Training Objective 33:55
The training objective is derived from the lower bound on $\log p_ heta(x_0)$, simplifying to minimizing a sum of terms ($\mathcal{L}_{t-1}$) that compares the mean predicted reconstruction ($\mu_{ heta, t}$) with the true process mean ($\mu_{ ilde{t}, t}$). This loss function is used to train the model to predict noise/reconstruct missing information at each time step.
Foundation Model Paradigm Shift 1:02:00
Modern LLMs utilize a paradigm shift involving pre-training on massive unlabelled data and adapting via zero-shot or few-shot learning, reducing the need for collecting extensive downstream labeled datasets. The goal is to build a general foundation that can be adapted to unlimited tasks.
Low-Rank Adaptation (LoRA) Efficiency 1:15:00
To adapt massive pre-trained models without changing all parameters ($\theta$), LoRA restricts the weight update matrix $\Delta W$ to a low rank ($A B$). This significantly reduces the number of trainable parameters and is critical for memory efficiency, especially when serving multiple users.

Watch on YouTube Full article

Jul 21, 2026 · 1:29:47

Causal Models Need Causal Data - Xaira’s X-Cell model (Bo Wang & Ci Chu)

Xaira Therapeutics introduced X-Cell, a novel 4.9-billion-parameter diffusion language model designed as a virtual cell foundation model of biology. The model is trained on X-Atlas/Pisces—a massive dataset spanning 25.6 million single cells across 16 biological contexts and generated via the Perturb-seq platform. The core breakthrough lies in shifting from descriptive (observational) data to causal (interventional) data, allowing the model to predict how a cell will respond to genetic perturbations it has never encountered. This capability is crucial for advancing drug discovery by moving beyond trial-and-error methods.

Key takeaways

Causality vs. Correlation in Biology 20:07
Observational atlases (descriptive data) can describe biology, but they are fundamentally underpowered to learn causality. To predict the outcome of an intervention (e.g., knocking down a gene), causal data—generated through high-throughput perturbation screens—is required.
X-Cell Architecture and Training 1:03:27
X-Cell utilizes a diffusion language model approach, which treats gene expression prediction as an iterative 'editing' process rather than an autoregressive one. This architecture allows it to generate high-dimensional transcriptomic data by refining noisy representations until they minimize loss against the ground truth.
Data Generation Scale and Engineering 1:16:47
The model is powered by Perturb-seq, a technique combining high-throughput CRISPR perturbation with single-cell RNA sequencing. This process generates massive 2D datasets (perturbation on one axis, gene expression on the other) across millions of cells while minimizing batch effects.
Generalization and Translational Potential 1:25:07
X-Cell demonstrated impressive generalization by accurately predicting perturbation responses in active T-cells, even when the model was only trained on resting T-cell data. This suggests the potential to predict novel biology in unseen contexts.

Watch on YouTube Full article