Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog
Traditional monitoring using the Golden Signals (Latency, Error, Traffic, Saturation) is insufficient for Generative AI (GenAI) applications. To ensure a healthy GenAI system, engineers must implement three additional monitoring layers: Cost, Safety, and Quality. Key strategies include granular cost attribution via multi-level tagging, monitoring for prompt injection and PII leakage, and tracking quality metrics like hallucination rate and RAG retrieval quality.
Key takeaways
-
GenAI Monitoring Shift
GenAI applications are non-deterministic (same prompt yields different responses), have variable cost structures, face new attack vectors, and produce subjectively judged output, requiring continuous quality evaluation in the live environment.
-
Cost Monitoring Imperatives
5:51
Track three major cost drivers: Token creep (uncontrolled context window expansion), Model drift (switching to more expensive models, e.g., Opus 4.8), and Uncached calls (up to 70% of spend can be redundant without effective caching).
-
Safety Monitoring
15:15
Implement guardrails to detect Prompt injection rate, PII detection rate (target 0% of production outputs), Content moderation score (toxicity/harm), and Jailbreak attempts (aiming for 100% block rate).
-
Quality Metrics
Focus on five quality metrics: Hallucination rate, Relevance score (using embedding similarity), User satisfaction (NPS/thumbs up), Answer completeness, and RAG retrieval quality (e.g., Top K accuracy).