SWE-bench Is Saturated: How Coding Models Actually Compare Now

For two years SWE-bench Verified was the thermometer for automated software engineering. It no longer is, and the reason is simple: the top three models on the board sit within one percentage point of each other. Once a benchmark compresses every contender into a single band, it has stopped measuring and started decorating.

Let us compare three models that represent three different bets: Claude Opus 5 (Anthropic, released 24 July 2026), GPT-5.6 Sol (OpenAI), and Kimi K3 (Moonshot AI, a 2.8 trillion parameter MoE with weights expected on 27 July).

The numbers

Three caveats before anyone reads that table as a ranking.

First: the sources disagree. Vals AI, which runs the minimal mini-swe-agent harness with bash as the only tool, puts Opus 5 at 97.0%. BenchLM, updated 24 July, puts it at 96%, with Claude Mythos 5 at 95.5% and Claude Fable 5 at 95%. LLM Stats has Fable 5 in front at 95.0%. Each figure is correct in its own context and all of them become meaningless the moment you mix them.

Second: the harness matters as much as the model. Kimi K3’s Terminal-Bench 2.1 result (88.3%) comes from Moonshot’s own launch table, running KimiCode at maximum reasoning. GPT-5.6 Sol’s 89.5% comes from Artificial Analysis, which runs the Terminus 2 harness in an e2b sandbox and reports pass@1 averaged over three repeats per task. A half point gap between those two numbers means nothing.

Third: SWE-bench Verified has a contamination problem. After OpenAI’s withdrawal in February 2026, several trackers now recommend looking at SWE-bench Pro instead, which uses actively maintained repositories with no public ground truth leakage. The scores drop hard: Claude Fable 5 leads at 80.3%, with Opus 5 behind at 79.2%. That is a seventeen point fall on essentially the same kind of task.

What this means in practice

The interesting gap is not between the three flagship models. It is between Verified and everything else.

On SWE-bench Verified, which asks for a patch to a well described GitHub issue in a familiar Python repository, we are above 95%. On SWE-bench Pro, using code the model has not seen, we are around 80%. On Terminal-Bench 2.1, which asks the model to install packages, configure servers, investigate broken builds and adjust course by reading command output, we are under 90% across just 89 tasks.

The distance between those three figures is the distance between “can write a patch” and “can do the job.” The first problem is solved. The second is not.

Which one to pick

Claude Opus 5 has the strongest profile on long horizon work: it leads Artificial Analysis’s Agentic Index at 55.3 and ARC-AGI-3 at 30.2%, roughly 3.75 times the next best model. It costs $5 and $25 per million tokens, half the price of Fable 5. Artificial Analysis flags the caveat itself: measured hallucination rate is high, which matters in any workflow where the agent has no way to verify its own claims.

GPT-5.6 Sol wins on command line work and on pure abstract reasoning. It is the sensible choice when the bottleneck is the shell environment rather than application code. Output tokens cost more.

Kimi K3 is the structural story. If the weights genuinely land on 27 July under a usable licence, it becomes the first downloadable model with independently verified frontier scores. Until then you reach it only through the API at $3 and $15, and it is slower: Artificial Analysis measures it at roughly 34 tokens per second on Moonshot’s API, with around seven seconds to first token.

The operational takeaway

Stop picking models off the SWE-bench Verified leaderboard. It is saturated and, according to several trackers, contaminated.

Look at three things instead: the score on SWE-bench Pro or Terminal-Bench 2.1 with the harness stated explicitly, the cost per task rather than per token, and how the model behaves when it is wrong. That last variable appears on no leaderboard, and it is the one that decides whether an agent saves you time or costs you time.