“But it is also well-known throughout the industry that benchmarks are often quite gameable.”
Our reading, the sentence that was checked:
AI benchmarks are often quite gameable.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, No, Virginia, AGI is not imminent, paragraph 8.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-no-virginia-agi-is-not-imminent-39aa05-a2
We recorded 17 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-08-20. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- benchmarks are often quite gameable
- well-known throughout the industry
Our interpretation
- Gameable means that shortcuts, dataset biases or leakage can produce strong scores without the intended capability; deliberate cheating is not required.
- Often means a recurring problem across established benchmarks and research areas, rather than a specified majority of all benchmarks.
- The context explicitly invokes research from 2019 and its continuing relevance four years later. Assess that cumulative period through publication.
- Industry awareness is assessed through published research recognition, not a literal survey of every practitioner.
Counts as supported if: Research documents recurring benchmark vulnerabilities across tasks or domains, including successful shortcuts that bypass the intended capability.
Counts as contradicted if: Broad controlled audits show that such vulnerabilities are rare or ineffective, and that strong scores generally require the intended capabilities.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
A 2020 review documents shortcut learning across research areas, including benchmark scores based on textures or correlated words rather than the intended abilities [e1]. A 2019 experiment shows good performance on SNLI and MNLI without reading the premise, while the same shortcut performs poorly on ANLI [e2]. These findings support recurring, publicly recognized benchmark vulnerabilities within the specified 2019–August 20, 2023 period [e1, e2].
Limitations
- The evidence does not estimate the proportion of all benchmarks affected or measure industry-wide awareness [e1, e2].
- ANLI's resistance to hypothesis-only shortcuts shows that this vulnerability is not universal [e2].
- These shortcut-learning findings do not establish deliberate gaming by developers [e1, e2].
Evidence (2 items)
Supports the claim: arXiv,
Shortcut learning currently occurs across deep learning, causing machines to fail unexpectedly.
Why it matters here: The review documents benchmark shortcuts across vision, language, reinforcement learning and algorithmic decision-making. This supports a recurring, publicly recognized problem rather than an isolated vulnerability.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: arXiv,
hypothesis-only models obtain good performance on SNLI and MNLI.
Why it matters here: Models obtain good scores on two established inference benchmarks while omitting the premise needed for the intended reasoning task. This directly demonstrates benchmark vulnerability to statistical shortcuts.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, No, Virginia, AGI is not imminent
But it is also well-known throughout the industry that benchmarks are often quite gameable.
- Where
- paragraph 8
- Published
- 2023-08-20
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-no-virginia-agi-is-not-imminent-39aa05-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-no-virginia-agi-is-not-imminent-39aa05-a2.