Agentic video understanding in Gemini
Summary
Agentic Video Understanding in Gemini addresses the high token cost and inefficiency of processing entire long videos by utilizing an agentic loop architecture. Instead of feeding the model the full video data (which can exceed 100,000 tokens), the system allows the model to intelligently use specialized tools—such as `get transcript` or `get frames`—to localize its analysis. This approach significantly reduces token consumption while improving accuracy by focusing attention only on relevant segments of the video.
Key takeaways
-
Inefficiency of Naive Video Processing
Processing an entire long video can generate over 100,000 tokens, making it computationally expensive and inefficient if the required information is localized to a small segment (e.g., frame 900).
-
Agentic Approach for Efficiency
The new pipeline does not require the entire video; instead, the model references the video and uses tools to extract only necessary data pieces (like transcripts or specific frames) before proceeding.
-
Agentic Loop Mechanism
2:00
Analysis follows a traditional agentic loop: Thinking $\rightarrow$ Acting (using tools) $\rightarrow$ Observing (receiving observations like extracted frames), and looping until the final answer is derived. This process reduces token cost and enhances performance by zooming in on critical video functions.
Technical details
-
Agentic Video Understanding
0s
This capability allows Gemini to intelligently break down a query against a long video using tools, rather than processing the entire input. It is available in Gemini 3.7 Flash, 3.6 Flash, and 3.5 Flash-Lite.
-
Tool Calling Functions
85s
The model can utilize specific functions to process video data: `get transcript` (for textual context), `get frames` (to specify desired frames and frames per second for localized visual analysis), and accessing audio information.
Mentioned resources
- Gemini
- Google Developers Resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.