# Multimodal Collaborative Agents for Next-Gen Commerce — Nidhi Kaushik Vyas, Google DeepMind

## Executive summary

The talk outlines a framework for multimodal collaborative agents designed to handle 'fuzzy intent' in commerce and consumer verticals. Instead of acting as simple search bar wrappers that assume well-defined user goals, these advanced agents proactively guide users who arrive with only a 'vibe.' The core mechanism is a three-stage loop—Discovery, Research, and Response—which systematically builds a working state from multimodal inputs (images, context) to determine the optimal next question or presentation format.

## Key takeaways

- Handling Fuzzy Intent: Agents must address the 'articulation gap,' recognizing that users often arrive with vague preferences rather than precise keywords. The agent's role is to proactively elicit and refine these fuzzy intents.
- The Collaborative Loop: The system operates in a loop: Discovery (building the working state), Research (determining the best way to ask/find information), and Response (adapting the output format).
- Prioritizing Information Gain: The agent must calculate which unknown variable, when queried, will yield the 'maximal information gain' to move the conversation forward efficiently (e.g., determining room width is critical before recommending furniture).
- Multimodal Elicitation: For subjective preferences, visual inspiration boards and multimodal inputs are significantly more effective than text-based questioning for establishing a common language between the user and the system.

## Technical details

- Working State Assembly (Discovery): The agent assembles a working state using past conversations, personal context, user queries, and reference images. It separates 'hard constraints' from 'soft constraints,' assigns a confidence score to each piece of information, and identifies variables requiring real-time refreshing (e.g., inventory) [4:45, 6:33].
- State Evaluation Metrics: The system uses auto-raters to ensure data integrity by checking for fact retention, confidence calibration within error bounds, and counterfactual sensitivity (testing if changing a query part changes only the relevant constraints) [7:28].
- Research Phase Strategy: This phase involves mapping known user constraints back to the product catalog ontology. The agent determines the optimal elicitation method (e.g., visual board vs. text) and monitors micro-signals like hovers or clicks to improve the confidence model [3:20].
- Adaptive Response Generation: The agent must adapt its output format based on the user's query type. Examples include using summary lists for policy questions, comparison tables for comparing products, or visual references for style inspiration [5:40]. Evaluation focuses on 'format accuracy' and 'user actionability.'

## Practical implications

- Design systems to accept 'vibes' (fuzzy intent) rather than requiring perfectly formed queries.
- Implement a multi-stage agent loop that prioritizes information gain over checklist completion.
- Integrate multimodal inputs and use visual/comparative formats for preference elicitation.
- Build robust auto-rating layers into the system to validate state integrity, confidence scores, and constraint sensitivity.

## Topics

Multimodal AI, Conversational Agents, Fuzzy Intent Recognition, System Architecture, Elicitation Strategy

Source: https://www.youtube.com/watch?v=AhQpRalYlyg