The Video Signal

Topic

DeepSWE Benchmark

All digests tagged DeepSWE Benchmark

Building Docs for Agents, Not Humans: Inside OpenWiki thumbnail

· 16:52

Building Docs for Agents, Not Humans: Inside OpenWiki

OpenWiki is an open-source Command Line Interface (CLI) designed to automatically generate and maintain repository documentation specifically optimized for consumption by coding agents. Unlike human-centric wikis, OpenWiki structures content into self-contained, highly searchable snippets using the Open Knowledge Format (OKF). It integrates deeply into a codebase via GitHub Actions, ensuring that documentation remains current with every code change while minimizing manual effort.

Key takeaways

  1. Built for Agents, Not Humans 5:04

    OpenWiki's core thesis is that since agents are increasingly writing code, the documentation must be structured for agent retrieval. This means content must consist of self-contained snippets with predictable headings and optimized context window usage, rather than long narrative pages.

  2. Automatic Maintenance via CI/CD

    The CLI facilitates automatic documentation updates by writing a GitHub Actions workflow. This action runs periodically (e.g., daily), checks the Git history, and uses an agent to generate or update the wiki based on code changes, minimizing manual intervention.

  3. Adoption of Open Knowledge Format (OKF) 11:48

    The system adopts OKF (Google's Open Knowledge Format) by adding a deterministic YAML front matter to every markdown file. This includes fields like `type`, `title`, and `description`, which significantly improves retrieval, filtering, and searching capabilities for agents.

  4. Performance Gains in Benchmarks 15:00

    Early evaluations using the DeepSWE benchmark show that OpenWiki usage leads to fewer tool calls, fewer searches, and a significant drop in token consumption for coding agents while maintaining or improving results.

Watch on YouTube Full article

DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve thumbnail

· 17:34

DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve

DeepSWE is a contamination-resistant benchmark featuring 113 original software engineering tasks designed to rigorously test large language model (LLM) capabilities in long-horizon coding problems. Unlike benchmarks that scrape existing pull requests (PRs), DeepSWE's methodology prevents models from cheating via training data leakage. The evaluation focuses on observable behavior and task completion, revealing significant performance gaps between top-tier models and others, particularly concerning the ability to handle multi-part prompts and self-verify code.

Key takeaways

  1. Contamination Resistance 2:03

    DeepSWE utilizes 113 tasks written from scratch, ensuring they were not scraped from existing PRs. This design choice prevents models from cheating by accessing solutions or discussions available in the wild (a major issue with benchmarks like SweetBench Pro) [0:00], [1:03].

  2. Focus on Observable Behavior

    The verifiers are designed to reward correct observable behavior rather than relying on specific, PR-derived implementations or private helper functions. This significantly reduces the risk of false negatives and positives [11:45].

  3. Model Failure Modes Observed 5:20

    Analysis revealed that while stronger models tend to test their own work (a positive sign), they can also exhibit forgetfulness when handling multi-part prompts, such as implementing one required version (e.g., synchronous) but dropping another (e.g., asynchronous) [5:18].

  4. Prompting Style for Realism 10:23

    DeepSWE prompts are designed to be high-level and ambiguous, mirroring real-world engineering tasks, rather than being overly verbose 'to-do lists' that prescribe a specific solution method [10:15].

Watch on YouTube Full article