DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve
DeepSWE is a contamination-resistant benchmark featuring 113 original software engineering tasks designed to rigorously test large language model (LLM) capabilities in long-horizon coding problems. Unlike benchmarks that scrape existing pull requests (PRs), DeepSWE's methodology prevents models from cheating via training data leakage. The evaluation focuses on observable behavior and task completion, revealing significant performance gaps between top-tier models and others, particularly concerning the ability to handle multi-part prompts and self-verify code.
Key takeaways
-
Contamination Resistance
2:03
DeepSWE utilizes 113 tasks written from scratch, ensuring they were not scraped from existing PRs. This design choice prevents models from cheating by accessing solutions or discussions available in the wild (a major issue with benchmarks like SweetBench Pro) [0:00], [1:03].
-
Focus on Observable Behavior
The verifiers are designed to reward correct observable behavior rather than relying on specific, PR-derived implementations or private helper functions. This significantly reduces the risk of false negatives and positives [11:45].
-
Model Failure Modes Observed
5:20
Analysis revealed that while stronger models tend to test their own work (a positive sign), they can also exhibit forgetfulness when handling multi-part prompts, such as implementing one required version (e.g., synchronous) but dropping another (e.g., asynchronous) [5:18].
-
Prompting Style for Realism
10:23
DeepSWE prompts are designed to be high-level and ambiguous, mirroring real-world engineering tasks, rather than being overly verbose 'to-do lists' that prescribe a specific solution method [10:15].