“I’m wary of putting too much stock into them, because they’re easily gamed, and quite often, they’re not an effective way of showing actual progress.”
Our reading, the sentence that was checked:
AI benchmarks are easily gamed and often fail to show actual progress.
Assessment: Unresolved Evidence as of
Where and when Ed Zitron wrote it
Where's Your Ed At, The Phony Comforts of AI Optimism, paragraph 15.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment wyea-optimistic-cowardice-966c32-a1
We recorded 21 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-03-24. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- the benchmarks used to test AI performance
- they’re easily gamed
- quite often
- not an effective way of showing actual progress
Our interpretation
- The context concerns benchmarks used to evaluate generative language models, rather than every benchmark in every AI field.
- Gaming means increasing a reported score through exposure to test material or test-specific optimization without corresponding general improvement.
- Actual progress means improved performance on independent tasks relevant to the capability being claimed.
- 'Quite often' means a recurring, consequential problem across commonly used evaluations; it does not mean all benchmarks or necessarily a majority.
- Because this describes an ongoing state without a period, assess benchmarking practice during March 2025 through the post date.
Counts as supported if: Evidence establishes readily achievable score inflation and recurring failures of benchmark gains to transfer to independent relevant tasks across commonly used evaluations.
Counts as contradicted if: Broad validation shows that commonly used evaluations resist readily achievable score inflation and that their gains reliably track independent relevant performance.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The literature gives substantial grounds for caution: contamination can inflate scores, and reviews document recurring evaluation problems. METR also shows that a carefully designed evaluation can track interpretable capability gains. These findings neither establish nor refute the full breadth of 'easily' and 'quite often' for the benchmarking landscape described in March 2025.
Limitations
- The earlier contamination studies are context, not period-matched outcome evidence.
- A review selected for shortcomings cannot provide an unbiased failure frequency.
- Evidence that some benchmarks work does not contradict a claim that others often fail.
Evidence (3 items)
Context only: arXiv; Jasper Dekoninck, Mark Niklas Müller and Martin Vechev,
However, data contamination can lead to inflated performance, rendering them unreliable for model comparison.
Why it matters here: ConStat reports contamination findings across several popular models and distinguishes inflated scores from generalizing performance. These earlier results establish a real failure mechanism, but not its frequency in the target period.
- Publisher
- arXiv; Jasper Dekoninck, Mark Niklas Müller and Martin Vechev
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Maria Eriksson and colleagues,
This paper presents an interdisciplinary meta-review of about 100 studies that discuss shortcomings in quantitative benchmarking practices, published in the last 10 years.
Why it matters here: The review documents recurring concerns including gaming, contamination and weak links between tests and the capabilities they purport to measure. Its selection of studies about shortcomings does not measure how often benchmark improvements fail to reflect real progress.
- Where
- Abstract
- Publisher
- arXiv; Maria Eriksson and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; METR researchers,
Furthermore, frontier AI time horizon has been doubling approximately every seven months since 2019, though the trend may have accelerated in 2024.
Why it matters here: This evaluation connects improvement to the duration of tasks measured against human performance, showing that benchmarks can provide interpretable evidence of capability gains. It does not establish the reliability of benchmarks generally, and the authors qualify transfer to real-world work.
- Where
- Abstract
- Publisher
- arXiv; METR researchers
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
A broad, period-matched audit of commonly used evaluations measuring gaming difficulty and whether score improvements transfer to independent relevant tasks.
Where it was said (1)
Said once in the sources this register covers.
: Where's Your Ed At, The Phony Comforts of AI Optimism
I’m wary of putting too much stock into them, because they’re easily gamed, and quite often, they’re not an effective way of showing actual progress.
- Where
- paragraph 15
- Published
- 2025-03-24
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment wyea-optimistic-cowardice-966c32-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.