# Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog

## Executive summary

Traditional monitoring using the Golden Signals (Latency, Error, Traffic, Saturation) is insufficient for Generative AI (GenAI) applications. To ensure a healthy GenAI system, engineers must implement three additional monitoring layers: Cost, Safety, and Quality. Key strategies include granular cost attribution via multi-level tagging, monitoring for prompt injection and PII leakage, and tracking quality metrics like hallucination rate and RAG retrieval quality.

## Key takeaways

- GenAI Monitoring Shift: GenAI applications are non-deterministic (same prompt yields different responses), have variable cost structures, face new attack vectors, and produce subjectively judged output, requiring continuous quality evaluation in the live environment.
- Cost Monitoring Imperatives: Track three major cost drivers: Token creep (uncontrolled context window expansion), Model drift (switching to more expensive models, e.g., Opus 4.8), and Uncached calls (up to 70% of spend can be redundant without effective caching).
- Safety Monitoring: Implement guardrails to detect Prompt injection rate, PII detection rate (target 0% of production outputs), Content moderation score (toxicity/harm), and Jailbreak attempts (aiming for 100% block rate).
- Quality Metrics: Focus on five quality metrics: Hallucination rate, Relevance score (using embedding similarity), User satisfaction (NPS/thumbs up), Answer completeness, and RAG retrieval quality (e.g., Top K accuracy).

## Technical details

- Cost Attribution Strategy: To gain granular cost visibility, implement mandatory tagging at four levels: Feature level (e.g., chat vs. summarization), User level (User ID/Org name for chargebacks), Model level (e.g., GPT-5.5, OpenAI), and Endpoint level (Production vs. Staging).
- RAG Quality Measurement: When using Retrieval Augmented Generation (RAG), track the quality of the retrieval process using metrics like Top K accuracy or normalized discounted gains, as document relevance is critical.
- LLM Observability: The overall monitoring stack must integrate Cost, Safety, and Quality metrics alongside traditional Golden Signals (Latency, Error, Traffic, Saturation).

## Practical implications

- Integrate continuous quality evaluation directly into the monitoring stack, moving beyond simple '200 OK' status codes.
- Establish a multi-level tagging strategy (Feature, User, Model, Endpoint) to accurately attribute and control GenAI spending.
- Set aggressive, measurable thresholds for safety metrics (e.g., 0% PII leakage, 100% jailbreak block rate) and quality metrics (e.g., >85% positive user feedback).
- Utilize specialized observability tools (like Datadog Agent Observability) to manage the complexity of GenAI monitoring.

## Topics

Generative AI, LLM Observability, Monitoring, Cost Management, AI Safety, RAG, Datadog Agent Observability, LLM Observability docs

Source: https://www.youtube.com/watch?v=rTojoVotlD8