Datadog Deleted All Its AI Context. It Worked.
Summary
Datadog detailed its journey scaling AI coding agents across 4,000 engineers, highlighting that performance improvements were achieved by deleting years of accumulated context files (context rot). The discussion emphasizes the critical role of building dedicated evaluation (evals) platforms to make data-driven decisions about model selection and agent capabilities. Key findings include using evals to replay historical PRs for code review guardrails and adapting hiring practices away from traditional LeetCode interviews toward real-world, large codebase tasks.
Key takeaways
-
Context Rot: Deleting Context Improved Performance
2:49
The team found that deleting old, accumulated AI context files (written prior to models like Sonnet 3.5) led to better evaluation scores, demonstrating 'context rot'—where historical information becomes irrelevant or harmful to the agent's performance.
-
Evals for Code Review and Regression Testing
5:49
The first concrete application of evals was building a platform that replays historical PRs known to have caused incidents, allowing agents to act as a last guardrail before production deployment.
-
Shift from Productivity to Ambition
59:02
The core lesson learned is that the goal of AI adoption should not solely be increasing productivity, but rather 'increasing ambition'—enabling teams to attempt and validate more complex ideas.
-
AI-Driven Interviewing
53:25
The process of hiring is evolving away from low-signal LeetCode interviews toward real-world scenarios that require AI to navigate and understand large, complex codebases.
Technical details
-
AI Agent Adoption Timeline
34s
Datadog initially launched a Proof of Concept (POC) using Cursor in early 2025, which quickly scaled from 100 to over 1,000 users. The second wave involved Claude Code, which saw high adoption among power users due to its ability to function effectively without an IDE open.
-
Eval Platform Architecture
241s
The evaluation system is a fully custom solution written in Go with sandboxing. It is designed to support multiple harnesses and models from different sources, making it model-agnostic.
-
Open-Weight Model Viability
316s
Based on evals, open-weight models are considered fully viable for highly repeatable processes (like code review), assuming they improve by approximately 50% from their current state. For simple tasks, cheaper open models can be used to assess regressions.
-
Evaluation Frequency and Integration
359s
Evals are run nightly (once a day). Ad-hoc runs are available for PRs specifically targeting evals or changes in steering documents. The goal is to identify apparent regressions rather than running on every context change.
Mentioned resources
- Datadog
- Cursor
- Claude Code
- GLM 5.2
- Tessl
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.