# DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve

## Executive summary

DeepSWE is a contamination-resistant benchmark featuring 113 original software engineering tasks designed to rigorously test large language model (LLM) capabilities in long-horizon coding problems. Unlike benchmarks that scrape existing pull requests (PRs), DeepSWE's methodology prevents models from cheating via training data leakage. The evaluation focuses on observable behavior and task completion, revealing significant performance gaps between top-tier models and others, particularly concerning the ability to handle multi-part prompts and self-verify code.

## Key takeaways

- Contamination Resistance: DeepSWE utilizes 113 tasks written from scratch, ensuring they were not scraped from existing PRs. This design choice prevents models from cheating by accessing solutions or discussions available in the wild (a major issue with benchmarks like SweetBench Pro) [0:00], [1:03].
- Focus on Observable Behavior: The verifiers are designed to reward correct observable behavior rather than relying on specific, PR-derived implementations or private helper functions. This significantly reduces the risk of false negatives and positives [11:45].
- Model Failure Modes Observed: Analysis revealed that while stronger models tend to test their own work (a positive sign), they can also exhibit forgetfulness when handling multi-part prompts, such as implementing one required version (e.g., synchronous) but dropping another (e.g., asynchronous) [5:18].
- Prompting Style for Realism: DeepSWE prompts are designed to be high-level and ambiguous, mirroring real-world engineering tasks, rather than being overly verbose 'to-do lists' that prescribe a specific solution method [10:15].

## Technical details

- Benchmark Scope and Scale: DeepSWE comprises 113 original tasks across multiple languages (Typescript, JavaScript, Python, Rust, Go), drawing from nearly 100 unique repositories. The average prompt length is significantly shorter than comparable benchmarks like SweetBench Pro [623].
- Verification Methodology: The verifiers prioritize checking the functional outcome (observable behavior) over adherence to specific internal naming conventions or private functions, making them more robust and less prone to false negatives [11:45].
- Agent Harnessing: The benchmark uses the mini-suite agent, an agent-agnostic harness, to focus on base model performance. The platform also supports bespoke creation of challenges by core contributors and maintainers [14:30].

## Practical implications

- The benchmark highlights the need for LLM evaluation systems to move beyond simple code generation and focus on verifiable, observable behavior in complex, multi-step tasks.
- For build engineers, DeepSWE suggests that future CI/CD pipelines integrating AI agents must incorporate anti-contamination measures (e.g., ensuring test cases are not derived from model training data) and robust verifiers that check functional output rather than structural adherence.

## Topics

LLM Benchmarking, Software Engineering Tasks, Contamination Resistance, Agentic Workflow Testing, Code Verification Systems, DeepSWE Benchmark, Data Curve

Source: https://www.youtube.com/watch?v=Yk87oUPVaxU