# What Is MLflow? Tracing AI Agents & LLM Workflows

## Executive summary

This video details the necessity of advanced observability for multi-agent and LLM workflows, arguing that traditional monitoring (HTTP status, response time) is insufficient for diagnosing complex AI failures. MLflow is presented as an OpenTelemetry-compatible platform that captures detailed traces, allowing engineers to monitor not just request success, but the internal logic, tool usage, and quality of the AI's decision path. Key techniques include using LLM judges for non-deterministic evaluation and integrating quality gates into the CI pipeline.

## Key takeaways

- Beyond Request-Level Monitoring: Multi-agent systems require visibility into internal failures like silent tool failures, cascading latency, context overflow, and non-determinism, which standard monitoring dashboards cannot detect. (0:00)
- MLflow's Core Primitive: The Trace: MLflow uses 'traces'—complete records of one request—composed of 'spans.' Each span details a single action (e.g., LLM call, DB query), capturing inputs, outputs, latency, and token counts. (0:47)
- Advanced Evaluation with LLM Judges: For non-deterministic AI, MLflow allows using a second model (LLM Judge) to grade agent output against criteria like tool call correctness, relevance, and safety. (2:32)
- Production Deployment Best Practices: For production, the tracking server must use a real database (e.g., Postgres/MySQL), asynchronous trace logging should be enabled, and sampling should be used to manage overhead. (4:10)

## Technical details

- MLflow Observability: MLflow is an OpenTelemetry compatible platform that captures inputs, outputs, and metadata from every step of an AI request, enabling proper LLM observability for multi-agent systems. (0:29)
- Trace Composition: A trace is composed of individual spans, which are organized into a parent-child tree structure. This structure tracks which component called which, along with specific inputs, outputs, and timing metrics. (0:55)
- LLM Judge Scoring: Engineers can use a second model to grade agent output against defined criteria (e.g., tool call correctness, efficiency, relevance, safety) to address non-determinism. (2:32)
- MLflow Integration: MLflow supports automatic logging via `mlflow.langchain.autolog` and custom code tracing using the decorator `@mlflow.trace`, ensuring all agent logic is captured automatically. (5:30)
- Backend Configuration: For production, the tracking server should point to a robust backend store like Postgres or MySQL, and asynchronous logging must be enabled to prevent trace export from adding latency to the user response. (4:10)

## Practical implications

- Implement MLflow to move beyond simple HTTP monitoring and gain visibility into the internal decision-making process of multi-agent systems.
- Treat LLM evaluation as a quality gate, integrating LLM Judge scoring and versioned runs into the CI pipeline, not just in notebooks.
- When deploying MLflow, ensure the tracking server uses a robust, scalable database (e.g., Postgres) and implements asynchronous logging to maintain low latency.
- Leverage OpenTelemetry compatibility to dual-export traces to existing platform backends like Jaeger or Grafana Tempo.

## Topics

MLflow, AI Observability, LLM, Multi-Agent Systems, OpenTelemetry, CI/CD, Tracing, AI Updates Newsletter

Source: https://www.youtube.com/watch?v=iZX6d0OdZys