From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI
The presentation outlines the shift from traditional Vision-Language Models (VLMs) and Vision-Action Models (VALAs) toward Embodied Foundation Models. The core goal is to create unified AI capable of perceiving, reasoning, and acting in the physical world. Key breakthroughs include addressing the massive data inefficiency of long video inputs (token waste) and managing context bloat using data sparse Mixture of Experts (MoE). A major finding is a new scaling law: joint training on perception, reasoning, and control allows a 10x reduction in expensive teleoperation data by increasing video pretraining data.
Key takeaways
-
Embodied Foundation Models (EFMs)
The north star is building physical AI foundations that unify perception, reasoning, and control into a single model, bridging the physical and digital world. This moves beyond the limitations of separate VLM, VLA, and World Model architectures.
-
Addressing Data Sparsity and Token Waste
5:40
When modeling long video (e.g., one hour), millions of visual tokens are generated, but ground truth is extremely sparse (e.g., only 2% of tokens are useful). The solution involves developing a 'natural perceptive objective' to automatically predict and focus on percepts that matter (e.g., a gripper tip) rather than treating all background pixels equally.
-
Context Management via Data Sparse MoE
9:01
To combat context bloat from always-on cameras, the approach utilizes a data sparse Mixture of Experts (MoE) architecture. This includes a router that determines per layer which tokens to read and which to skip, allowing the model to focus compute on task-relevant information.
-
New Scaling Law for Robotics
9:01
Jointly training on perception, reasoning, and control yields a significant scaling law: 10x more video pretraining data can substitute for 10x less expensive teleoperation (teleop) data, drastically lowering the cost of embodied AI.