# SWE-bench Scores: The Gap Between Passing Tests and Shipping Software
**Plutonous** | November 9, 2026 | 6 min read
> What SWE-bench measures, why OpenAI retired Verified, and how task selection, test quality and agent harnesses change a coding score.
Tags: Coding Agents, SWE-bench, AI Evaluation, Software Engineering, Developer Tools, Benchmarks, Enterprise AI, Model Procurement
---
*Cover: Generated editorial illustration of a small testing fixture beside a larger production system. It is a conceptual comparison, not evidence that a patch is correct.*
**TL;DR: SWE-bench began with 2,294 issues from 12 Python repositories; Verified later selected 500 human-reviewed tasks.[1][2] OpenAI's February 23, 2026 audit flagged material issues in 59.4% of 138 deliberately selected, frequently failed tasks, not 59.4% of the full benchmark.[3] A coding score is evidence about a task set, grader and agent configuration, not a direct percentage of a software team's work.**
The coding leaderboard is a distribution channel. A single percentage travels easily through a launch post, procurement deck and developer conversation. The contract behind it travels less easily: which tasks, which environment, how many attempts and what the tests actually accept.
This analysis revisits the original October 2023 SWE-bench, its August 2024 Verified release and subsequent evaluation work. It does not rank the latest November 2026 coding models. The business question is how to translate benchmark evidence into a purchasing decision without silently changing what the number means.
### Why This Matters Now
An engineering team buys help with its own repositories, review standards and deployment constraints. The useful benchmark is one that predicts those outcomes closely enough to guide a controlled trial.
## The Denominator: Five Hundred Tasks Are a Defined Population
SWE-bench pairs issue descriptions with repository states and evaluates proposed patches against tests. Its original population was 2,294 problems from 12 Python repositories.[1] That was a substantial advance over asking for an isolated function. It still defines a particular population rather than the entire software profession.
Verified's 500-task subset was released on August 13, 2024 after human review to improve problem clarity and test validity.[2] Filtering changes the denominator and the mixture of work. A percentage on Full, Lite or Verified is therefore a different measurement, even when a model name stays the same.
On an illustrative 500-task evaluation, one additional resolved task moves the score by 0.2 percentage points. That arithmetic is exact; its interpretation requires more information. A two-point gain could reflect ten additional successes concentrated in one library, or improvements spread across many repositories. A buyer with a different language and task mix should not assume either pattern transfers.
The practical response is to request task-level outcomes, failure categories and a stable evaluation manifest. A comparison becomes much more useful when the reader can inspect which problems changed rather than treating the aggregate as an attribute permanently attached to the model.
## The Grader: Passing Tests Is a Specific Claim
The Verified release explains two checks: tests that should begin failing and then pass after the fix, and tests that should continue passing.[2] Together they test requested behavior and some regression coverage. They cannot establish every property an engineering organization cares about, such as maintainability or compatibility beyond the tested environment.
OpenAI's February 2026 audit examined 138 tasks that its o3 model did not consistently solve across 64 runs. It reported flawed tests or descriptions in 59.4% of this failure-selected group. OpenAI is both the reporting provider and a participant in model competition; this is its audit, not an independently reproduced census.[3]
### Issue categories in OpenAI's selected 138-task audit
Provider-reported audit
Percentages reported by OpenAI on February 23, 2026. The sample was selected from tasks o3 did not consistently solve, not randomly from all 500 Verified tasks.
- label: Tests enforce unspecified implementation details; value: 35.5; displayValue: 35.5%
- label: Tests demand additional unspecified functionality; value: 18.8; displayValue: 18.8%
- label: Other material issues; value: 5.1; displayValue: 5.1%
100
0%
100%
Share of 138 audited tasks
Source: OpenAI's own audit, not independently replicated here. Categories total 59.4% using the reported rounded values. Selection on repeated model failures prevents extrapolation to the complete benchmark or to other models.
The real story isn't that all testing is broken. It is that a grader has a specification, and that specification needs maintenance. A narrowly written test can reject an acceptable implementation. A permissive suite can miss a defect. Neither outcome licenses replacing tests with subjective impressions; it calls for independent checks on consequential claims.
## The Harness: A Score Belongs to an Entire System
SWE-agent's May 2024 research made interface design part of the experimental object. Navigation, editing and execution tools affect how effectively a language model can work in a repository.[4] The model is an important component, but a score produced by an agent system is not a clean measurement of weights alone.
A reproducible comparison needs the same dataset revision, task identifiers, base commits, container images, agent version, tool permissions, attempt policy and resource budget. It should disclose excluded tasks, empty patches, infrastructure failures and whether the reported result is one attempt or a selection from multiple candidates. This article makes no inference-speed comparison because those conditions are not normalized here.
The maintainers' FAQ distinguishes submitted, completed, resolved and error counts. It also warns that reused run identifiers can retrieve cached outcomes even after a prediction patch changes.[8] That operational detail captures the broader issue: an evaluation result depends on execution bookkeeping as well as model behavior. Buyers need the audit trail, not just a screenshot.
## Freshness: Replace Exposure Risk Without Inventing Comparability
SWE-rebench, first submitted in May 2025, describes continuous collection of new repository tasks to address the exposure risk of a static public benchmark.[5] The approach is useful because a public issue and its eventual human fix can enter training data. Freshness reduces one source of uncertainty; it does not automatically make tasks clear or graders complete.
LiveCodeBench's March 2024 work uses newly collected contest problems and evaluates several coding capabilities.[6] It provides another way to investigate exposure, but contest programming and repository maintenance are different jobs. A lower score on a fresher test is not, by itself, a measured contamination penalty. Task difficulty and coverage may have changed too.
The original September 2025 SWE-Bench Pro paper broadens repository and task coverage, including held-out and commercial settings.[7] Its existence reinforces a sensible evaluation portfolio: test several relevant populations, retain each test's boundaries and avoid drawing one universal ranking from incompatible task sets. A benchmark name without a revision is insufficient for a serious comparison.
## Procurement: Buy Verified Work at a Known Review Cost
The useful local pilot starts with recent, representative tasks that the supplier has not been trained or tuned against where feasible. Keep the acceptance criteria independent of the generated patch. Record elapsed time, review time, task completion, regressions and the cost of unsuccessful attempts. Include difficult cases that a team might otherwise omit because they make the demonstration less attractive.
A developer should also inspect how the agent handles uncertainty. Does it identify missing requirements, preserve an unrelated behavior and report a test it could not run? Those are operational outcomes. They belong alongside resolved-task rate because they determine whether review becomes faster or merely moves to a different stage.
### A selected audit is not a population estimate
Do not quote OpenAI's 59.4% as the defect rate of all SWE-bench Verified tasks. It applies to 138 tasks selected for repeated failures. Keep the selection rule attached to the number everywhere it appears.
Let's be clear: software benchmarks have helped make coding progress measurable. Their strategic value increases when suppliers disclose the conditions under which scores were produced and buyers test the connection to their own work. The product worth paying for is a change that survives review at a predictable cost.
## Sources
1. [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://arxiv.org/abs/2310.06770)
2. [Introducing SWE-bench Verified](https://openai.com/index/introducing-swe-bench-verified/)
3. [Why SWE-bench Verified no longer measures frontier coding capabilities](https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/)
4. [SWE-agent: Agent-Computer Interfaces](https://arxiv.org/abs/2405.15793)
5. [SWE-rebench](https://arxiv.org/abs/2505.20411)
6. [LiveCodeBench](https://arxiv.org/abs/2403.07974)
7. [SWE-Bench Pro](https://arxiv.org/abs/2509.16941)
8. [SWE-bench FAQ](https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/docs/faq.md)
*Last updated: November 9, 2026*
---
*Source: [LLM Rumors](https://www.llmrumors.com/news/swe-bench-score-production-software-gap)*