“Basically, beating benchmarks doesn’t tell you as much as you might think. Often when you do, it’s a statistical artifact.”
Our reading, the sentence that was checked:
Beating AI benchmarks is often a statistical artifact and says less than people assume.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, No, Virginia, AGI is not imminent, paragraph 10.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-no-virginia-agi-is-not-imminent-5387d8-a1
We recorded 17 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-08-20. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- beating benchmarks doesn’t tell you as much as you might think
- Often when you do, it’s a statistical artifact
Our interpretation
- Statistical artifact means that apparent success materially depends on dataset-specific correlations or labeling conventions instead of the capability being inferred.
- In context, beating benchmarks includes strong or leading benchmark performance, not only formally exceeding a human score.
- Often means recurring documented cases across benchmarks. It does not mean that most gains are entirely illusory.
- The context expressly compares the 2019 literature with its continuing relevance in 2023, so the target is that cumulative period.
- The claim about what readers think is assessed as a warning against overinterpreting scores, not as a measured claim about public beliefs.
Counts as supported if: Controlled evaluations repeatedly connect strong benchmark performance to shortcuts or dataset artifacts, with weaker performance when those artifacts are removed or challenged.
Counts as contradicted if: Broad controlled evaluations show that apparent successes generally survive removal of suspected artifacts and that artifact-driven success is exceptional.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Controlled language and vision studies show that strong scores can materially reflect shortcuts or labeling biases. This supports the contextual warning that apparent benchmark success repeatedly overstates the intended capability. The finding depends on reading often qualitatively and artifact as a material contribution, not a claim that most progress is fictitious. Fresh-test results also show genuine transferable gains.
Limitations
- No representative estimate establishes how often this happens across all AI benchmarks.
- Artifact-driven gains and genuine progress can coexist.
- A stronger reading that most benchmark victories are wholly spurious is not established.
Evidence (3 items)
Supports the claim: arXiv,
the heuristics targeted by HANS.
Why it matters here: Models scoring well on MNLI failed controlled examples that invalidated their syntactic shortcuts. The experiment links apparent benchmark competence to heuristics that do not reliably perform the intended inference.
- Where
- Figure 1 caption
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: arXiv,
significantly higher accuracy of 71.1%.
Why it matters here: For the desktop-computer category, this exceeds the study's approximately 30% unbiased-label baseline. The wider relabeling analysis attributes a significant part of ImageNet progress to labeling biases, with only partial transfer to reassessed labels.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: Proceedings of Machine Learning Research,
larger gains on the new test sets.
Why it matters here: Fresh CIFAR-10 and ImageNet tests preserve meaningful comparative progress despite lower absolute accuracy. This limits a sweeping dismissal of benchmark gains, but does not refute the existence of recurring artifact-driven successes.
- Where
- Abstract
- Publisher
- Proceedings of Machine Learning Research
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, No, Virginia, AGI is not imminent
Basically, beating benchmarks doesn’t tell you as much as you might think. Often when you do, it’s a statistical artifact.
- Where
- paragraph 10
- Published
- 2023-08-20
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-no-virginia-agi-is-not-imminent-5387d8-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-no-virginia-agi-is-not-imminent-5387d8-a1.