A few years ago, a model topping a major benchmark leaderboard was a reasonably strong signal that it was genuinely more capable across a broad range of tasks. That correlation has weakened noticeably, and the compressed 2026 release cycle, which has already produced seven major frontier model launches in a single quarter, has made the problem worse rather than better, since leaderboard rankings now turn over on a timescale too fast for independent, careful verification to keep pace.

Goodhart's Law, applied to AI benchmarks

The core dynamic is a version of a well-known pattern: once a measure becomes a target, it stops being a reliable measure. When labs know a specific benchmark is widely used to compare models publicly, there's strong incentive to specifically optimize for that benchmark, sometimes through legitimate general capability improvements that happen to show up there, and sometimes through training choices narrowly targeted at that specific benchmark's format and question types, which improves the score without proportionally improving general real-world capability.

Data contamination: a harder problem than it sounds

A specific, well-documented version of this problem is training-data contamination. Benchmark questions or very similar variants of them ending up, deliberately or accidentally, in a model's training data, inflating its benchmark score in a way that doesn't reflect genuine capability on novel problems. This is genuinely difficult to fully prevent given how much public text exists discussing popular benchmarks, and it's a big part of why newer, more carefully controlled or private benchmark sets tend to show smaller gaps between models than older, more widely circulated public benchmarks do.

Benchmarks measure a narrower slice of capability than they imply

Even an uncontaminated benchmark score reflects performance on a specific, bounded task format (often multiple-choice questions or short structured problems) which is a meaningfully narrower measure than "general intelligence" or "overall capability," despite how leaderboard rankings are often reported and discussed. A model can score well on a benchmark's specific question format while performing less reliably on open-ended, ambiguous, real-world tasks that don't share that format, which is exactly the gap between benchmark performance and practical usefulness that shows up repeatedly across the tool categories we cover, including in our test of AI writing assistants on long-form work, where benchmark-strong models still show real structural weaknesses on a realistic, extended task.

What the compressed release cycle is doing to the problem

When a new frontier model launches roughly every two weeks, as the pace through early 2026 has approached, independent evaluators simply don't have time to run the kind of careful, methodologically rigorous benchmarking that would meaningfully validate a lab's own claimed results before the next release supersedes it in the news cycle. That's a structural, not incidental, problem. It means the public's primary source of comparative model information, at any given moment, is disproportionately weighted toward each lab's own self-reported numbers, precisely the numbers most subject to the optimization pressure Goodhart's Law describes.

Why this doesn't mean benchmarks are worthless

None of this means benchmarks provide no useful signal, a benchmark score is still meaningfully more informative than no evidence at all, and benchmarks focused on narrower, harder-to-game domains (rigorous coding benchmarks with held-out test cases, for instance) have generally held up better than broad, widely-circulated general knowledge benchmarks. The right response to benchmark fatigue isn't abandoning benchmarks. It's weighting them appropriately as one input among several, rather than as a decisive, standalone verdict, and being especially cautious about benchmark claims from a release too recent for independent verification to have caught up.

What to actually check instead

For anyone evaluating a model for a real use case, the more reliable approach is testing the model directly on a representative sample of your actual task, the same evaluation discipline we recommend throughout our tool coverage. See our guide to choosing AI tools for the broader framework. Benchmark scores are a reasonable first filter for narrowing a field of candidates; they're a poor substitute for testing against your specific, real task before committing to a model or tool.