From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI
Summary
The presentation outlines the shift from traditional Vision-Language Models (VLMs) and Vision-Action Models (VALAs) toward Embodied Foundation Models. The core goal is to create unified AI capable of perceiving, reasoning, and acting in the physical world. Key breakthroughs include addressing the massive data inefficiency of long video inputs (token waste) and managing context bloat using data sparse Mixture of Experts (MoE). A major finding is a new scaling law: joint training on perception, reasoning, and control allows a 10x reduction in expensive teleoperation data by increasing video pretraining data.
Key takeaways
-
Embodied Foundation Models (EFMs)
The north star is building physical AI foundations that unify perception, reasoning, and control into a single model, bridging the physical and digital world. This moves beyond the limitations of separate VLM, VLA, and World Model architectures.
-
Addressing Data Sparsity and Token Waste
5:40
When modeling long video (e.g., one hour), millions of visual tokens are generated, but ground truth is extremely sparse (e.g., only 2% of tokens are useful). The solution involves developing a 'natural perceptive objective' to automatically predict and focus on percepts that matter (e.g., a gripper tip) rather than treating all background pixels equally.
-
Context Management via Data Sparse MoE
9:01
To combat context bloat from always-on cameras, the approach utilizes a data sparse Mixture of Experts (MoE) architecture. This includes a router that determines per layer which tokens to read and which to skip, allowing the model to focus compute on task-relevant information.
-
New Scaling Law for Robotics
9:01
Jointly training on perception, reasoning, and control yields a significant scaling law: 10x more video pretraining data can substitute for 10x less expensive teleoperation (teleop) data, drastically lowering the cost of embodied AI.
Technical details
-
Model Architectures
123s
The speaker distinguishes between VLMs (image/video + text $\rightarrow$ text), ER models (spatial understanding/grounding points), VALAs (outputting actions), World Models (outputting video), and Semantic World Models (learning useful representations). The proposed EFM unifies all these capabilities.
-
Data Efficiency and Loss Function
340s
Traditional loss functions are problematic because they are either too sparse, too synthetic, or too indiscriminate. The proposed solution focuses on predicting 'percepts that will matter' automatically, moving beyond hardcoded percepts.
-
Agentic Task Reframing
541s
Classical Computer Vision (CV) tasks, such as object detection, are reframed as agentic tasks. The model must decide to tile the image, raise the contrast, and propose bounding boxes sequentially to locate an object (e.g., a bird).
-
Robustness and Fusion
1130s
Joint perceptive and control modeling significantly increases robustness compared to pure VLA models. For instance, changing the background of a table will cause a pure policy to fail, but the joint model maintains robustness.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.