DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve
Summary
DeepSWE is a contamination-resistant benchmark featuring 113 original software engineering tasks designed to rigorously test large language model (LLM) capabilities in long-horizon coding problems. Unlike benchmarks that scrape existing pull requests (PRs), DeepSWE's methodology prevents models from cheating via training data leakage. The evaluation focuses on observable behavior and task completion, revealing significant performance gaps between top-tier models and others, particularly concerning the ability to handle multi-part prompts and self-verify code.
Key takeaways
-
Contamination Resistance
2:03
DeepSWE utilizes 113 tasks written from scratch, ensuring they were not scraped from existing PRs. This design choice prevents models from cheating by accessing solutions or discussions available in the wild (a major issue with benchmarks like SweetBench Pro) [0:00], [1:03].
-
Focus on Observable Behavior
The verifiers are designed to reward correct observable behavior rather than relying on specific, PR-derived implementations or private helper functions. This significantly reduces the risk of false negatives and positives [11:45].
-
Model Failure Modes Observed
5:20
Analysis revealed that while stronger models tend to test their own work (a positive sign), they can also exhibit forgetfulness when handling multi-part prompts, such as implementing one required version (e.g., synchronous) but dropping another (e.g., asynchronous) [5:18].
-
Prompting Style for Realism
10:23
DeepSWE prompts are designed to be high-level and ambiguous, mirroring real-world engineering tasks, rather than being overly verbose 'to-do lists' that prescribe a specific solution method [10:15].
Technical details
-
Benchmark Scope and Scale
145s
DeepSWE comprises 113 original tasks across multiple languages (Typescript, JavaScript, Python, Rust, Go), drawing from nearly 100 unique repositories. The average prompt length is significantly shorter than comparable benchmarks like SweetBench Pro [623].
-
Verification Methodology
The verifiers prioritize checking the functional outcome (observable behavior) over adherence to specific internal naming conventions or private functions, making them more robust and less prone to false negatives [11:45].
-
Agent Harnessing
The benchmark uses the mini-suite agent, an agent-agnostic harness, to focus on base model performance. The platform also supports bespoke creation of challenges by core contributors and maintainers [14:30].
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.