Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind
Summary
The talk outlines a framework for multimodal collaborative agents designed to handle 'fuzzy intent' in commerce and consumer verticals. Instead of acting as simple search bar wrappers that assume well-defined user goals, these advanced agents proactively guide users who arrive with only a 'vibe.' The core mechanism is a three-stage loop—Discovery, Research, and Response—which systematically builds a working state from multimodal inputs (images, context) to determine the optimal next question or presentation format.
Key takeaways
-
Handling Fuzzy Intent
1:48
Agents must address the 'articulation gap,' recognizing that users often arrive with vague preferences rather than precise keywords. The agent's role is to proactively elicit and refine these fuzzy intents.
-
The Collaborative Loop
3:23
The system operates in a loop: Discovery (building the working state), Research (determining the best way to ask/find information), and Response (adapting the output format).
-
Prioritizing Information Gain
13:41
The agent must calculate which unknown variable, when queried, will yield the 'maximal information gain' to move the conversation forward efficiently (e.g., determining room width is critical before recommending furniture).
-
Multimodal Elicitation
5:20
For subjective preferences, visual inspiration boards and multimodal inputs are significantly more effective than text-based questioning for establishing a common language between the user and the system.
Technical details
-
Working State Assembly (Discovery)
445s
The agent assembles a working state using past conversations, personal context, user queries, and reference images. It separates 'hard constraints' from 'soft constraints,' assigns a confidence score to each piece of information, and identifies variables requiring real-time refreshing (e.g., inventory) [4:45, 6:33].
-
State Evaluation Metrics
728s
The system uses auto-raters to ensure data integrity by checking for fact retention, confidence calibration within error bounds, and counterfactual sensitivity (testing if changing a query part changes only the relevant constraints) [7:28].
-
Research Phase Strategy
320s
This phase involves mapping known user constraints back to the product catalog ontology. The agent determines the optimal elicitation method (e.g., visual board vs. text) and monitors micro-signals like hovers or clicks to improve the confidence model [3:20].
-
Adaptive Response Generation
540s
The agent must adapt its output format based on the user's query type. Examples include using summary lists for policy questions, comparison tables for comparing products, or visual references for style inspiration [5:40]. Evaluation focuses on 'format accuracy' and 'user actionability.'
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.