# Datadog Deleted All Its AI Context. It Worked.

## Executive summary

Datadog detailed its journey scaling AI coding agents across 4,000 engineers, highlighting that performance improvements were achieved by deleting years of accumulated context files (context rot). The discussion emphasizes the critical role of building dedicated evaluation (evals) platforms to make data-driven decisions about model selection and agent capabilities. Key findings include using evals to replay historical PRs for code review guardrails and adapting hiring practices away from traditional LeetCode interviews toward real-world, large codebase tasks.

## Key takeaways

- Context Rot: Deleting Context Improved Performance: The team found that deleting old, accumulated AI context files (written prior to models like Sonnet 3.5) led to better evaluation scores, demonstrating 'context rot'—where historical information becomes irrelevant or harmful to the agent's performance.
- Evals for Code Review and Regression Testing: The first concrete application of evals was building a platform that replays historical PRs known to have caused incidents, allowing agents to act as a last guardrail before production deployment.
- Shift from Productivity to Ambition: The core lesson learned is that the goal of AI adoption should not solely be increasing productivity, but rather 'increasing ambition'—enabling teams to attempt and validate more complex ideas.
- AI-Driven Interviewing: The process of hiring is evolving away from low-signal LeetCode interviews toward real-world scenarios that require AI to navigate and understand large, complex codebases.

## Technical details

- AI Agent Adoption Timeline: Datadog initially launched a Proof of Concept (POC) using Cursor in early 2025, which quickly scaled from 100 to over 1,000 users. The second wave involved Claude Code, which saw high adoption among power users due to its ability to function effectively without an IDE open.
- Eval Platform Architecture: The evaluation system is a fully custom solution written in Go with sandboxing. It is designed to support multiple harnesses and models from different sources, making it model-agnostic.
- Open-Weight Model Viability: Based on evals, open-weight models are considered fully viable for highly repeatable processes (like code review), assuming they improve by approximately 50% from their current state. For simple tasks, cheaper open models can be used to assess regressions.
- Evaluation Frequency and Integration: Evals are run nightly (once a day). Ad-hoc runs are available for PRs specifically targeting evals or changes in steering documents. The goal is to identify apparent regressions rather than running on every context change.

## Practical implications

- Implement automated, data-driven evaluation pipelines (evals) that run against historical production incidents (replayed PRs) to establish a robust code review guardrail.
- Shift focus from general productivity metrics to 'ambition' and the ability to validate complex, multi-stage hypotheses using AI POCs.
- Design agentic workflows to solve surrounding annoyances (e.g., linting errors, formatting issues) rather than attempting to rewrite core business logic, as these tasks are easier for open-weight models.
- Establish a formal process for monitoring and mitigating 'context rot' by regularly running evals that test the current model against historical context.

## Topics

AI Agentic Workflows, Evaluation Engineering (Evals), Code Review Automation, Large Language Models (LLMs), Software Development Lifecycle (SDLC), Datadog, Cursor, Claude Code, GLM 5.2, Tessl

Source: https://www.youtube.com/watch?v=ru5SX4eonzM