Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber
Uber detailed its approach to building closed-loop evaluation systems for a multimodal agent designed to enhance food photography on Uber Eats. The system addresses the challenge of improving image quality at massive scale (90 billion run rate/year) while preserving merchant authenticity and avoiding homogenization. The architecture involves an end-to-end pipeline incorporating Image Understanding/Routing agents, iterative Image Editing agents, multi-dimensional QA gates, and sophisticated continuous learning loops that detect drift and auto-tune the system without human intervention.
Key takeaways
-
Closed-Loop System Design
17:07
The system operates on a closed feedback loop: production data is sampled, compared against human labels (golden source of truth), mismatches are diagnosed by an 'Umbrella Diagnosis Agent,' and the auto-tuning pipeline adjusts agent configurations. This process ensures continuous improvement without requiring constant human intervention.
-
Handling Model Drift
20:03
To prevent static models from failing in a live environment, the system uses production data sampling to detect drift. The 'Diagnoser' agent localizes the issue and triggers an auto-tuning pipeline that benchmarks against the golden dataset before deployment.
-
Evaluation Metrics for Agents
21:05
For routing agents, evaluation involves measuring precision and recall using a confusion matrix (potentially an N x N matrix if multiple branching paths exist). For the enhancement process, 'Pass at K' measures the pass rate after K iterations of refinement.