“However, benchmark success is a limited indicator of general intelligence.”
Our reading, the sentence that was checked:
Benchmark success is only a limited indicator of general intelligence.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Rumors of AGI’s arrival have been greatly exaggerated, paragraph 19.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-rumors-of-agis-arrival-have-been-0779c8-a3
We recorded 19 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2026-02-17. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- benchmark success
- limited indicator of general intelligence
Our interpretation
- Plain reading: published evidence shows that high scores on AI benchmarks often do not carry over to robust performance when tasks change modestly, or reflect overfitting to the benchmark rather than general quality. So benchmark scores alone are a weak guide to general intelligence.
- He describes an ongoing state with no period. Evaluator uses research published from 2024 up to the post date of 2026-02-17.
Counts as supported if: Credible studies before the post show benchmark scores overstating general ability, for example performance drops under small perturbations, or leaderboard rankings reflecting overfitting rather than general quality.
Counts as contradicted if: Credible evidence that benchmark success reliably tracks robust general capability under novel conditions, with little gap between benchmark and perturbed or real-world performance.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The claim is that benchmark success is only a limited indicator of general intelligence. The contract looks at research from 2024-01-01 to the post date of 2026-02-17. Two studies in that period fit its support condition. The GSM-Symbolic study (Mirzadeh et al., first posted 2024-10-07, listed as ICLR 2025) reports that the performance of all models it tested declined when only the numbers in GSM8K-style questions were changed. It also reports drops of up to 65% when one clause was added that seemed relevant but did not affect the answer. 'The Leaderboard Illusion' (Singh et al., preprint, 2025-04-29) estimates, conservatively, that even limited extra Chatbot Arena data can give relative gains of up to 112% on the arena distribution. The authors state that the dynamics they describe result in 'overfitting to Arena-specific dynamics rather than general model quality'. The packet contains no evidence that benchmark success reliably tracks robust general capability.
Limitations
- 'General intelligence' has no agreed measure, and neither study measures it directly. The studies show benchmark scores overstating robustness, or being shaped by effects specific to one benchmark.
- The evidence covers two settings, GSM8K-style maths questions and the Chatbot Arena leaderboard. It does not cover benchmarks in general.
- The Leaderboard Illusion is a preprint. Its 112% figure is the authors' own estimate of gains on the arena distribution. The overfitting statement is their interpretation and is not a separate measurement of general quality.
- 'Limited indicator' has no threshold, so the claim is partly conceptual.
Evidence (2 items)
Supports the claim: arXiv / ICLR 2025 (Mirzadeh et al., Apple),
the performance of all models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark.
Why it matters here: Strong GSM8K scores did not hold up under trivial changes. So the benchmark score overstated robust reasoning.
- Where
- Abstract
- Publisher
- arXiv / ICLR 2025 (Mirzadeh et al., Apple)
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Supports the claim: arXiv (Singh et al., Cohere Labs and others),
Together, these dynamics result in overfitting to Arena-specific dynamics rather than general model quality.
Why it matters here: On the most-cited leaderboard, rank gains partly reflected benchmark-specific overfitting, not general quality. This shows benchmark success is a limited indicator.
- Where
- Abstract
- Publisher
- arXiv (Singh et al., Cohere Labs and others)
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Rumors of AGI’s arrival have been greatly exaggerated
However, benchmark success is a limited indicator of general intelligence.
- Where
- paragraph 19
- Published
- 2026-02-17
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-rumors-of-agis-arrival-have-been-0779c8-a3: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-rumors-of-agis-arrival-have-been-0779c8-a3.
HeldBy this site
The final-text check failed; overclaims: On the most-cited leaderboard, rank gains partly reflected benchmark-specific overfitting, not general quality..
About assessment marcus-rumors-of-agis-arrival-have-been-0779c8-a2, which was not published.