# MCPs for Observability Stacks

## Executive summary

This session details how MCP servers enhance traditional observability stacks by integrating AI capabilities for proactive system management. By correlating metrics, logs, traces, and events, MCPs allow engineers to move beyond reactive monitoring. Key features include automated anomaly detection (using techniques like setting business boundaries), natural language querying, and the use of 'skills'—reusable playbooks that guide AI agents through complex tasks such as root cause analysis, metric cleanup, and model selection for time series forecasting.

## Key takeaways

- Shift to Proactive Observability: The goal of modern observability is to move from reactive incident response to proactive anomaly detection, aiming to reduce Mean Time To Resolution (MTTR) by correlating telemetry across the entire stack.
- MCP's Role in Analysis: MCP servers enable AI agents to query and correlate data, automating root cause analysis. This capability replaces manual dashboard navigation and complex query writing using natural language prompts.
- Advanced Anomaly Detection: Anomaly detection identifies unusual patterns (spikes or drops) that deviate from expected behavior. Accurate detection requires defining 'business boundaries' to provide necessary context for the model.
- Automated Workflow and Model Selection: MCPs can use specialized 'skills' (reusable playbooks) to perform complex tasks. For instance, an AI assistant can analyze a query's time series characteristics and recommend switching between forecasting models (e.g., from Prophet to IMADS online).

## Technical details

- Observability Stack Integration: The system integrates metrics, logs, traces, and events into a unified context. The architecture layers anomaly detection on top of traditional observability stacks.
- Anomaly Detection Mechanics: Detection involves comparing original data against model predictions (the trend) and highlighting deviations using defined business boundaries, rather than relying solely on static thresholds.
- Querying and Data Sources: The demonstration utilized VictoriaMetrics as the backend and employed `lossql` for querying log data to count error spans.
- AI Agent Capabilities (Skills): MCP servers utilize 'skills'—reusable playbooks—to perform specific actions like metric catalog analysis, finding unused metrics, and generating alert configurations in YAML format.

## Practical implications

- Build engineers can implement automated monitoring pipelines that detect resource leaks or performance degradation (e.g., unused metrics) before they cause production incidents.
- The system facilitates faster root cause analysis by allowing natural language interaction with complex, multi-source observability data.
- Automated alert generation and model validation streamline the process of maintaining robust CI/CD monitoring dashboards.

## Topics

Observability, AI Assistants, Anomaly Detection, DevOps Monitoring, VictoriaMetrics, Agentic AI Foundation (AAIF)

Source: https://www.youtube.com/watch?v=aQHg6db9wRs