# Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber

## Executive summary

Uber detailed its approach to building closed-loop evaluation systems for a multimodal agent designed to enhance food photography on Uber Eats. The system addresses the challenge of improving image quality at massive scale (90 billion run rate/year) while preserving merchant authenticity and avoiding homogenization. The architecture involves an end-to-end pipeline incorporating Image Understanding/Routing agents, iterative Image Editing agents, multi-dimensional QA gates, and sophisticated continuous learning loops that detect drift and auto-tune the system without human intervention.

## Key takeaways

- Closed-Loop System Design: The system operates on a closed feedback loop: production data is sampled, compared against human labels (golden source of truth), mismatches are diagnosed by an 'Umbrella Diagnosis Agent,' and the auto-tuning pipeline adjusts agent configurations. This process ensures continuous improvement without requiring constant human intervention.
- Handling Model Drift: To prevent static models from failing in a live environment, the system uses production data sampling to detect drift. The 'Diagnoser' agent localizes the issue and triggers an auto-tuning pipeline that benchmarks against the golden dataset before deployment.
- Evaluation Metrics for Agents: For routing agents, evaluation involves measuring precision and recall using a confusion matrix (potentially an N x N matrix if multiple branching paths exist). For the enhancement process, 'Pass at K' measures the pass rate after K iterations of refinement.

## Technical details

- System Architecture: Image Enhancement Pipeline: The process begins with an Image Understanding/Routing agent (multimodality input) that determines if enhancement is needed. If yes, the image goes to an iterative Image Editing agent, which receives feedback from a QA agent. The final output passes through a post-processing and QA step before publication.
- Routing Agent Evaluation: The router uses structured outputs derived from multimodal inputs (text description, metadata, image) to grade against pass/fail criteria. Evaluation is treated as a classifier problem, measuring metrics like precision and recall.
- Auto-Tuning Pipeline: The tuning process involves the 'Reflect Agent' (identifying systemic issues in mismatches) and the 'Synthesize Agent' (updating the agent config based on feedback). The new configuration is registered in an agent store, completing a fully automated, no-human-in-the-loop cycle.
- QA Gate and Iterative Enhancement: The enhancement process includes multiple QA gates (e.g., plating, faithfulness, colors). If a gate fails, the feedback is pushed back to the prompt generator for another attempt, measuring 'Pass at K'—the pass rate after K iterations.

## Practical implications

- Implementing a multi-stage, closed-loop MLOps pipeline for generative AI agents.
- Designing robust evaluation metrics (e.g., Pass at K) that account for iterative refinement and failure modes.
- Building automated drift detection systems using production data sampling to maintain model performance over time.
- Structuring logging (JSON format) across all micro-agents in an orchestration flow to facilitate comprehensive diagnosis.

## Topics

Multimodal AI, Generative Agents, MLOps, Closed-Loop Evaluation, Image Processing, System Scaling

Source: https://www.youtube.com/watch?v=31GUkCBD-Uc