Robotics — The Video Signal

Sep 25, 2026 · 1:38:06

Runway’s Bet Beyond Video: World Models, Robotics, and the Neural OS — Anastasis Germanidis

Runway is evolving beyond generative video to build 'Interface World Models' and general-purpose world simulators. The core thesis is that scaling video models is sufficient to learn physics and human dynamics, making them suitable for robotics and simulating complex software interfaces. The company emphasizes that the ultimate goal is a fully neural operating system, where the interface itself is generated by the model, rather than relying on traditional code like HTML/CSS.

Key takeaways

World Models as the Endgame 1:30:20
The ultimate goal is a fully neural operating system where the model delivers the application end-to-end, generating both the language model output and the rendered pixels/interface. This shifts the focus from content creation to general-purpose world simulation.
Scaling Video Models for Physics 1:14:40
The belief is that if scaling laws apply to language models (LLMs), they will also apply to video. By scaling up video models, the system will inherently learn to simulate physics, human actions, and dynamics, making the model a general simulator.
Third-Person Video Data Advantage 1:25:50
The most plentiful source of data for training robotics models is third-person video data (observing others perform tasks), which is far more abundant than teleoperation or egocentric data. Video pre-training allows models to generalize to new environments and tasks.
The Importance of Counterfactual Generation 1:29:10
A key difference between standard video models and true world models is the ability to generate counterfactuals—simulating 'what if' scenarios (e.g., scoring a goal vs. failing to score a goal). This is critical for robust robotics training.

Watch on YouTube Full article

Sep 24, 2026 · 26:42

Robot Demos Are Easy. Reliability Is Hard — Jason Ma, Dyna Robotics

Dyna Robotics focuses on developing highly robust, generalist robotic policies for commercial deployment, arguing that high reliability is more critical than high performance in demos. The company utilizes a 'research and deployment flywheel' to build foundation models, achieving a 99.4% success rate in complex tasks like napkin folding over 24 hours. Key technical advancements include a 'pre-training data pyramid' (over 200,000 hours) and the use of reward models for scalable supervision, allowing the system to detect and recover from errors in long-horizon tasks.

Key takeaways

The Reliability Gap in Robotics 10:10
Achieving a high success rate (e.g., 99.4%) over extended periods (24 hours) is necessary for commercial viability, as standard models often stall at 80–90% success rates, making repeated tasks highly improbable.
The Research and Deployment Flywheel 2:00
Dyna Robotics combines frontier research with active commercial deployments to gather high-quality data, which informs and sharpens the focus of their model development, ensuring the product solves real-world problems.
Scalable Error Recovery via Reward Models 18:59
Instead of relying on manual oversight, the team developed reward models that score the robot's progress during complex tasks. Dips in this score signal a mistake, enabling targeted data collection and a human-in-the-loop active learning cycle for robust error recovery.
Generalization Across Sites
The model architecture is designed to generalize, allowing deployment at new customer sites (e.g., a laundromat, Red Bull events) without requiring site-specific fine-tuning or additional data.

Watch on YouTube Full article

Sep 24, 2026 · 46:27

When Will AI Make Me Scrambled Eggs? I Went To NVIDIA To Find Out.

The video details the shift in AI from Large Language Models (LLMs) generating text to World Models (WMs) that generate physical actions and simulations. NVIDIA, through its Cosmos Lab, is building WMs to enable physical AI in complex domains like robotics, self-driving cars, and factory automation. The Cosmos 3 platform fuses world understanding, world simulation, and action capability into a single model, allowing developers to test and verify policies in a simulated environment before real-world deployment. The core architectural components include a reasoner, a generator, and an action module.

Key takeaways

World Models vs. LLMs
While LLMs are symbolic and semantic (dealing with text), World Models are designed to produce direct actions (e.g., issuing guidance to a robot arm) and model the physical world. WMs allow developers to simulate complex physical scenarios (like a factory floor) without needing to build thousands of physical prototypes.
The Cosmos 3 Architecture 16:30
The Cosmos platform integrates three key components into a single model: world understanding (interpreting the physical state), world simulation (predicting how the world changes), and action capability (generating physical commands). This unified approach is critical for physical AI applications.
Scaling and Deployment 19:17
WMs are designed to operate in real-time, necessitating models of different sizes (e.g., Super Nano and Nano). The architecture supports a mix of deployment environments—from embedded devices (like Jetson or Dig Spark) to powerful data centers—to balance performance and computational constraints.
Verifiable Reward and Simulation
A major advantage of WMs is the ability to perform policy verification in simulation. This allows engineers to test safety and performance (e.g., for self-driving cars) across thousands of edge cases, dramatically accelerating development velocity compared to physical testing.

Watch on YouTube Full article

Sep 24, 2026 · 28:18

Robotics Has Been Stuck for 70 Years — Deepak Pathak, Skild AI

Deepak Pathak argues that robotics progress has stalled for approximately 70 years because the field has been treated as a hardware problem rather than a general intelligence problem. He introduces the concept of 'omni-bodied intelligence'—a single brain model applicable to any robot and any task, regardless of hardware. This approach leverages a 'data flywheel' that combines highly scalable data (simulation, human video) with high-quality, low-volume data (teleoperation) and, critically, real-world deployment data. Demonstrations include complex tasks like AirPods insertion, omelet cooking on $4,000 arms, and robust GPU assembly for NVIDIA's factory, showcasing the system's ability to handle real-world disturbances and zero-shot transfers.

Key takeaways

Robotics Stagnation and the General Brain 5:57
Robotics has historically been limited by approaching it as a hardware problem. The field is constrained by the lack of a general brain, leading to the 'Moravec's paradox' (what is easy for humans is hard for machines, and vice versa).
The Data Bottleneck 9:02
Collecting robot data via teleoperation is extremely slow and expensive. To reach the data scale of models like GPT-3, the entire US population would take over a century, highlighting the need for scalable data sources.
Omni-bodied Intelligence and the Data Flywheel 12:02
The proposed solution is an 'omni-bodied brain': one model for any robot and any task. This system utilizes a data flywheel, pre-training on scalable data (simulation, human video), post-training on teleoperation, and continuous improvement via deployment data.
Real-World Deployment and Robustness
The system demonstrates extreme robustness, performing tasks like GPU assembly in a randomized, noisy factory environment, and adapting to disturbances (e.g., recovering movement after disabling legs) without explicit mapping or planning.

Watch on YouTube Full article

Sep 23, 2026 · 20:42

From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI

The presentation outlines the shift from traditional Vision-Language Models (VLMs) and Vision-Action Models (VALAs) toward Embodied Foundation Models. The core goal is to create unified AI capable of perceiving, reasoning, and acting in the physical world. Key breakthroughs include addressing the massive data inefficiency of long video inputs (token waste) and managing context bloat using data sparse Mixture of Experts (MoE). A major finding is a new scaling law: joint training on perception, reasoning, and control allows a 10x reduction in expensive teleoperation data by increasing video pretraining data.

Key takeaways

Embodied Foundation Models (EFMs)
The north star is building physical AI foundations that unify perception, reasoning, and control into a single model, bridging the physical and digital world. This moves beyond the limitations of separate VLM, VLA, and World Model architectures.
Addressing Data Sparsity and Token Waste 5:40
When modeling long video (e.g., one hour), millions of visual tokens are generated, but ground truth is extremely sparse (e.g., only 2% of tokens are useful). The solution involves developing a 'natural perceptive objective' to automatically predict and focus on percepts that matter (e.g., a gripper tip) rather than treating all background pixels equally.
Context Management via Data Sparse MoE 9:01
To combat context bloat from always-on cameras, the approach utilizes a data sparse Mixture of Experts (MoE) architecture. This includes a router that determines per layer which tokens to read and which to skip, allowing the model to focus compute on task-relevant information.
New Scaling Law for Robotics 9:01
Jointly training on perception, reasoning, and control yields a significant scaling law: 10x more video pretraining data can substitute for 10x less expensive teleoperation (teleop) data, drastically lowering the cost of embodied AI.

Watch on YouTube Full article

Sep 11, 2026 · 40:58

Seattle DGX Spark Hackathon Winners Spotlight

This summary covers the NVIDIA DGX Spark Hackathon winners, spotlighting two advanced local AI applications: Kerberos, a shared spatial-awareness system for search-and-rescue (SAR) teams, and VELA, a voice-first, consent-controlled healthcare action system. Both projects demonstrate the power of running complex, multi-agent AI workflows entirely on local hardware (NVIDIA GB10), ensuring data privacy and real-time action capability.

Key takeaways

Local AI for Privacy and Reliability 22:40
Both winning systems (Kerberos and VELA) are designed to run entirely on the NVIDIA GB10, ensuring that sensitive data (e.g., medical records, live camera feeds) remains local and is not transmitted to the cloud, addressing critical security concerns in healthcare and SAR.
Agentic Workflows for Complex Tasks 25:40
The projects utilize multi-agent architectures (e.g., VELA's system) where specialized agents (like Parakeet for speech recognition, Neatron for reasoning, and Magpie for speech output) collaborate to perform complex, multi-step tasks, moving beyond simple chatbots to actionable outcomes.
Shared Situational Awareness in SAR 3:40
Kerberos creates a shared live map for SAR, integrating data from multiple sources (drones, robots, body cameras) to track responders, map searched areas, and pinpoint casualties or hazards, even indoors where GPS fails.

Watch on YouTube Full article

Aug 29, 2026 · 17:23

Tell the Robot What You Want — Sandhya Subramani, AWS

The presentation demonstrates 'Scout,' a rover running on a Raspberry Pi, that has been upgraded from a traditionally programmed robot into an agentic system using AWS's open-source framework, `strands`. This architecture allows the robot to understand and execute complex commands in natural language (e.g., counting people or responding to arbitrary prompts) by having an AI agent orchestrate calls to existing hardware policies/tools. The system utilizes a multi-agent setup (thinker, communication, voice) and operates on a hybrid cloud/edge model for both training (VLA models) and fast runtime execution.

Key takeaways

Agentic Robotics Architecture 2:07
By implementing an agent layer above existing movement policies, the robot can perform tasks it was never explicitly trained for. The agent decides *what* to do, while the policy determines *how* to execute the action.
Multi-Agent System Design 5:46
Scout runs three simultaneous agents: a thinker (constantly assessing the environment), a communication agent (via Telegram/web app), and an optional voice agent, allowing for complex, concurrent decision-making.
Hybrid Cloud/Edge Deployment 11:57
The system is designed to operate on a hybrid model: training (VLA models) occurs in the cloud using `agentcore`, while execution and policy calls happen rapidly at the edge (on the robot's local hardware).

Watch on YouTube Full article

Aug 12, 2026 · 57:04

Locknote: Reasons to be Cheerful: 0, 1, 2 - Emmz Rendle - NDC Copenhagen 2026

The talk presents a highly optimistic overview of technological progress across multiple domains, arguing that advancements in AI, quantum computing, and biotechnology are solving complex global problems. While addressing concerns about job displacement and climate change, the speaker emphasizes that these technologies—such as specialized LLMs (Gemma 4, Qwen 3.6) for code generation or CRISPR for personalized medicine—are tools that enhance human capability rather than replace it entirely. The presentation also touches on hardware advancements like shared memory architectures and post-quantum cryptography standards.

Key takeaways

AI in Diagnostics and Genomics 5:20
AI models are revolutionizing medicine by analyzing complex data sets, such as MRI scans (sub-millimeter slice images) to flag issues for radiologists. Furthermore, DNA/RNA analysis can detect hereditary disease risks and enable custom per-patient cancer treatments.
Quantum Computing Threat and Solution 12:30
The advent of powerful quantum computers (e.g., those capable of running Shor's algorithm) will render current classical cryptography (RSA, Diffie-Hellman, EDCSA) obsolete. The industry is responding by adopting post-quantum cryptography standards approved by NIST, specifically mentioning Dilithium and Kyber.
AI for Software Productivity 17:30
LLMs are not replacing programmers but acting as a new programming paradigm. They can generate functional code that is safe to put into production, allowing developers to focus on engineering the system's logic rather than manual typing.
Advancements in Robotics and Automation 23:20
Robotics are advancing through LLM-driven operating systems (like Gemini Robotics 1.5) that allow robots to reason through steps necessary for tasks, enabling scalable deployment across multiple units.
Sustainable Energy and Climate Solutions 31:40
New technologies include solar windows (generating electricity while visible) and 'solar canals,' which use floating panels over irrigation channels to prevent water evaporation and generate power.

Watch on YouTube Full article

Jul 25, 2026 · 21:45

Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google

The deployment of AI on edge devices (robotics, mobile phones) is constrained primarily by DRAM cost, necessitating the development of tiny LLMs. The talk details strategies—including aggressive quantization and fine-tuning—to shrink models like Gemma to run efficiently on low-power hardware (e.g., Raspberry Pi, Qualcomm NPUs). Tiny models (<500M parameters) are ideal for wide-scale deployment but often require specialized tasks (ASR, vision, function calling) and robust synthetic data generation for fine-tuning.

Key takeaways

DRAM Cost is the Primary Edge Constraint 1:45
The main challenge in edge AI deployment is not compute power, but DRAM cost. Mobile phone manufacturers are reducing on-device RAM, making model size critical. (3:25)
Quantization Enables Small Footprints 6:08
Models can be aggressively quantized (e.g., Gemma to 2.9 bits per weight) and optimized using techniques like per-layer embeddings to minimize memory footprint, allowing deployment on devices with limited RAM. (6:08)
Tiny Models Require Fine-Tuning 14:20
While small models are easy to use via zero-shot prompting, achieving high reliability for specific tasks like function calling or voice dictation requires fine-tuning using synthetic data sets. (14:28)
Voice-to-Function Calling is Key 16:00
Fine-tuning a small model to handle both ASR and function calling allows for robust, offline interaction on low-tier IoT/edge devices, which is crucial where complex UIs are difficult. (14:28)

Watch on YouTube Full article