# From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI

## Executive summary

The presentation outlines the shift from traditional Vision-Language Models (VLMs) and Vision-Action Models (VALAs) toward Embodied Foundation Models. The core goal is to create unified AI capable of perceiving, reasoning, and acting in the physical world. Key breakthroughs include addressing the massive data inefficiency of long video inputs (token waste) and managing context bloat using data sparse Mixture of Experts (MoE). A major finding is a new scaling law: joint training on perception, reasoning, and control allows a 10x reduction in expensive teleoperation data by increasing video pretraining data.

## Key takeaways

- Embodied Foundation Models (EFMs): The north star is building physical AI foundations that unify perception, reasoning, and control into a single model, bridging the physical and digital world. This moves beyond the limitations of separate VLM, VLA, and World Model architectures.
- Addressing Data Sparsity and Token Waste: When modeling long video (e.g., one hour), millions of visual tokens are generated, but ground truth is extremely sparse (e.g., only 2% of tokens are useful). The solution involves developing a 'natural perceptive objective' to automatically predict and focus on percepts that matter (e.g., a gripper tip) rather than treating all background pixels equally.
- Context Management via Data Sparse MoE: To combat context bloat from always-on cameras, the approach utilizes a data sparse Mixture of Experts (MoE) architecture. This includes a router that determines per layer which tokens to read and which to skip, allowing the model to focus compute on task-relevant information.
- New Scaling Law for Robotics: Jointly training on perception, reasoning, and control yields a significant scaling law: 10x more video pretraining data can substitute for 10x less expensive teleoperation (teleop) data, drastically lowering the cost of embodied AI.

## Technical details

- Model Architectures: The speaker distinguishes between VLMs (image/video + text $\rightarrow$ text), ER models (spatial understanding/grounding points), VALAs (outputting actions), World Models (outputting video), and Semantic World Models (learning useful representations). The proposed EFM unifies all these capabilities.
- Data Efficiency and Loss Function: Traditional loss functions are problematic because they are either too sparse, too synthetic, or too indiscriminate. The proposed solution focuses on predicting 'percepts that will matter' automatically, moving beyond hardcoded percepts.
- Agentic Task Reframing: Classical Computer Vision (CV) tasks, such as object detection, are reframed as agentic tasks. The model must decide to tile the image, raise the contrast, and propose bounding boxes sequentially to locate an object (e.g., a bird).
- Robustness and Fusion: Joint perceptive and control modeling significantly increases robustness compared to pure VLA models. For instance, changing the background of a table will cause a pure policy to fail, but the joint model maintains robustness.

## Practical implications

- Significantly reduces the cost and data requirements for training embodied AI by leveraging video pretraining data over expensive teleoperation data.
- Enables complex, multi-step physical tasks (e.g., sorting books by reading titles) that require deep integration between perception and control.
- Provides a pathway for building highly robust robotic systems that can adapt to environmental changes (e.g., lighting, background changes).

## Topics

Embodied AI, Foundation Models, Robotics, Multimodal AI, Computer Vision, Transformer Architectures, Armen Aghajanyan, Perceptron AI

Source: https://www.youtube.com/watch?v=ZZcE0HeO-Hc