In 2024 the question was “can the model solve the problem?” In 2026 the question has become “does the model know when it cannot?” It is a less exciting shift than the first one, and a far more consequential one for anyone putting an LLM into production.
Take the three models currently fighting over the reasoning crown: GPT-5.6 Sol (OpenAI), Claude Opus 5 (Anthropic) and Gemini 3.1 Pro (Google DeepMind).
The reasoning numbers
| Benchmark | GPT-5.6 Sol | Claude Opus 5 | Gemini 3.1 Pro |
|---|---|---|---|
| GPQA Diamond (Artificial Analysis) | 94.1% (max) | 93.7% (adaptive, high effort) | 94.1% (Preview) |
| Humanity’s Last Exam | 47.2% (Price Per Token, 23 Jul) | around 44.4% | 46.44 on the Scale SEAL board |
| ARC-AGI-2 (BenchLM, 25 Jul) | 92.5% | 90.4% | not in the top three |
| ARC-AGI-3 (LLM Stats) | not tracked | 30.2%, first of 4 models | not tracked |
| Human reference | GPQA: PhD experts at 65% | ARC-AGI-2: average individual 66% | HLE: no human baseline |
On GPQA Diamond the three models sit within four tenths of a point of each other, and all three are nearly thirty points above PhD holders working in their own field. On ARC-AGI-2, the benchmark that was supposed to hold out longest, GPT-5.6 Sol is at 92.5% against 66% for the average individual and a grand prize threshold set at 85%. That one is closing too.
The only test bed still genuinely open is ARC-AGI-3, the interactive version. Claude Opus 5 leads at 30.2%, roughly 3.75 times the next best model according to ARC Prize. But only four models have been evaluated, so treat that as a signal rather than a ranking.
The problem the scores do not show
Alongside accuracy on Humanity’s Last Exam, Scale SEAL publishes a second column: calibration error. It measures how far a model’s stated confidence drifts from its actual accuracy.
The figures are uncomfortable. Gemini 3.1 Pro Preview: 46.44% accuracy, 51% calibration error. GPT-5.4 Pro: 44.32% accuracy, 38% error. Meta’s Muse Spark: 40.56% accuracy, 50% error. Kimi K2.5: 24.37% accuracy, 67% error.
In plain terms: on the hardest problems we know how to write, these models are wrong more than half the time and do not notice. Artificial Analysis measures Claude Opus 5’s hallucination rate at around 50% and places it below Fable 5 on its AA-Omniscience metric for that reason, even though Opus 5 leads the overall Intelligence Index at 61.
A model that knows it does not know can be routed, verified, or queued for human review. A model that is confident and wrong is a much more expensive engineering problem.
When sources contradict each other
One case is worth pausing on, because it shows exactly how leaderboards get misread.
On FrontierMath, OpenAI’s GPT-5.6 launch post reports 89% on Tiers 1 to 3 and 83% on Tier 4. Epoch AI, which maintains the private set, verified GPT-5.5 Pro at 39.6% on Tier 4. Those two numbers describe different worlds. This is not necessarily dishonesty: the model differs, the compute budget differs, the set differs (public versus private), and the scoring rules differ. But if you lift the blog post figure and paste it next to Epoch’s in a slide, you are comparing two things with nothing in common.
The same applies to Humanity’s Last Exam. A FutureHouse investigation published in July 2025 estimated that around 30% of the reference answers for text-only chemistry and biology questions could be wrong. The benchmark team partially replicated the finding and announced a continuous revision process. A 47% score on a test where one answer in three may be incorrect deserves careful handling.
How to evaluate a reasoning model today
Three criteria, in decreasing order of usefulness.
One: calibration error, not accuracy. If your use case involves human review, a less accurate but well calibrated model generates less work than a more accurate and overconfident one. Scale SEAL is currently one of the few sources publishing the figure at all.
Two: cost per task, not per token. Artificial Analysis’s Intelligence Index v4.1 reports time, cost and output tokens for a single task. The spread is wide: from roughly a minute and a half per task on Grok 4.3 to thirteen and a half minutes on Claude Sonnet 4.6 at maximum effort. On a real workload that difference dominates everything else.
Three: benchmark freshness. GPQA Diamond is effectively saturated, and several aggregators have already demoted it to an unweighted reference. ARC-AGI-2 will follow within months. A score on a saturated benchmark carries no information.
Raw reasoning is now cheap and available from almost everyone. What is still not for sale is a model that can say “I do not know.”
Leave a Reply