Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog
Summary
Traditional monitoring using the Golden Signals (Latency, Error, Traffic, Saturation) is insufficient for Generative AI (GenAI) applications. To ensure a healthy GenAI system, engineers must implement three additional monitoring layers: Cost, Safety, and Quality. Key strategies include granular cost attribution via multi-level tagging, monitoring for prompt injection and PII leakage, and tracking quality metrics like hallucination rate and RAG retrieval quality.
Key takeaways
-
GenAI Monitoring Shift
GenAI applications are non-deterministic (same prompt yields different responses), have variable cost structures, face new attack vectors, and produce subjectively judged output, requiring continuous quality evaluation in the live environment.
-
Cost Monitoring Imperatives
5:51
Track three major cost drivers: Token creep (uncontrolled context window expansion), Model drift (switching to more expensive models, e.g., Opus 4.8), and Uncached calls (up to 70% of spend can be redundant without effective caching).
-
Safety Monitoring
15:15
Implement guardrails to detect Prompt injection rate, PII detection rate (target 0% of production outputs), Content moderation score (toxicity/harm), and Jailbreak attempts (aiming for 100% block rate).
-
Quality Metrics
Focus on five quality metrics: Hallucination rate, Relevance score (using embedding similarity), User satisfaction (NPS/thumbs up), Answer completeness, and RAG retrieval quality (e.g., Top K accuracy).
Technical details
-
Cost Attribution Strategy
456s
To gain granular cost visibility, implement mandatory tagging at four levels: Feature level (e.g., chat vs. summarization), User level (User ID/Org name for chargebacks), Model level (e.g., GPT-5.5, OpenAI), and Endpoint level (Production vs. Staging).
-
RAG Quality Measurement
When using Retrieval Augmented Generation (RAG), track the quality of the retrieval process using metrics like Top K accuracy or normalized discounted gains, as document relevance is critical.
-
LLM Observability
The overall monitoring stack must integrate Cost, Safety, and Quality metrics alongside traditional Golden Signals (Latency, Error, Traffic, Saturation).
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.