Datadog Deleted All Its AI Context. It Worked.
Datadog detailed its journey scaling AI coding agents across 4,000 engineers, highlighting that performance improvements were achieved by deleting years of accumulated context files (context rot). The discussion emphasizes the critical role of building dedicated evaluation (evals) platforms to make data-driven decisions about model selection and agent capabilities. Key findings include using evals to replay historical PRs for code review guardrails and adapting hiring practices away from traditional LeetCode interviews toward real-world, large codebase tasks.
Key takeaways
-
Context Rot: Deleting Context Improved Performance
2:49
The team found that deleting old, accumulated AI context files (written prior to models like Sonnet 3.5) led to better evaluation scores, demonstrating 'context rot'—where historical information becomes irrelevant or harmful to the agent's performance.
-
Evals for Code Review and Regression Testing
5:49
The first concrete application of evals was building a platform that replays historical PRs known to have caused incidents, allowing agents to act as a last guardrail before production deployment.
-
Shift from Productivity to Ambition
59:02
The core lesson learned is that the goal of AI adoption should not solely be increasing productivity, but rather 'increasing ambition'—enabling teams to attempt and validate more complex ideas.
-
AI-Driven Interviewing
53:25
The process of hiring is evolving away from low-signal LeetCode interviews toward real-world scenarios that require AI to navigate and understand large, complex codebases.