Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber
Summary
Uber detailed its approach to building closed-loop evaluation systems for a multimodal agent designed to enhance food photography on Uber Eats. The system addresses the challenge of improving image quality at massive scale (90 billion run rate/year) while preserving merchant authenticity and avoiding homogenization. The architecture involves an end-to-end pipeline incorporating Image Understanding/Routing agents, iterative Image Editing agents, multi-dimensional QA gates, and sophisticated continuous learning loops that detect drift and auto-tune the system without human intervention.
Key takeaways
-
Closed-Loop System Design
17:07
The system operates on a closed feedback loop: production data is sampled, compared against human labels (golden source of truth), mismatches are diagnosed by an 'Umbrella Diagnosis Agent,' and the auto-tuning pipeline adjusts agent configurations. This process ensures continuous improvement without requiring constant human intervention.
-
Handling Model Drift
20:03
To prevent static models from failing in a live environment, the system uses production data sampling to detect drift. The 'Diagnoser' agent localizes the issue and triggers an auto-tuning pipeline that benchmarks against the golden dataset before deployment.
-
Evaluation Metrics for Agents
21:05
For routing agents, evaluation involves measuring precision and recall using a confusion matrix (potentially an N x N matrix if multiple branching paths exist). For the enhancement process, 'Pass at K' measures the pass rate after K iterations of refinement.
Technical details
-
System Architecture: Image Enhancement Pipeline
430s
The process begins with an Image Understanding/Routing agent (multimodality input) that determines if enhancement is needed. If yes, the image goes to an iterative Image Editing agent, which receives feedback from a QA agent. The final output passes through a post-processing and QA step before publication.
-
Routing Agent Evaluation
640s
The router uses structured outputs derived from multimodal inputs (text description, metadata, image) to grade against pass/fail criteria. Evaluation is treated as a classifier problem, measuring metrics like precision and recall.
-
Auto-Tuning Pipeline
1085s
The tuning process involves the 'Reflect Agent' (identifying systemic issues in mismatches) and the 'Synthesize Agent' (updating the agent config based on feedback). The new configuration is registered in an agent store, completing a fully automated, no-human-in-the-loop cycle.
-
QA Gate and Iterative Enhancement
1140s
The enhancement process includes multiple QA gates (e.g., plating, faithfulness, colors). If a gate fails, the feedback is pushed back to the prompt generator for another attempt, measuring 'Pass at K'—the pass rate after K iterations.
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.