“And of course advancing on benchmarks isn’t itself enough anyway; few benchmarks capture the complexity of the real world. Even if LLMs could max out on all existing benchmark, we might still have a long way to go.”
Our reading, the sentence that was checked:
Advancing on benchmarks is not enough by itself because few benchmarks capture the complexity of the real world, and even if LLMs could max out all existing benchmarks, we might still have a long way to go.
Assessment: Can’t be checked Not checked against evidence
Where and when Gary Marcus wrote it
Marcus on AI, Evidence that LLMs are reaching a point of diminishing returns - and what that might mean, paragraph 24.
- Said
- Source observed
- live page, full text, public
- Evidence
- Not checkedboth screens found it cannot be checked
- Assessment recorded
- assessment marcus-evidence-that-llms-are-reaching-a-eac443-a1
We recorded 20 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-04-13. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- even if LLMs could max out on all existing benchmarks (qualifier, stated by the speaker)
- 'might still have a long way to go' is hedged (qualifier, stated by the speaker)
From the speaker
- even if LLMs could max out on all existing benchmarks
- 'might still have a long way to go' is hedged
Our interpretation
None added.
This claim is not graded on evidence. The record checks that the words are quoted correctly and attributed to the right person, and that the claim is classified correctly.
Current assessment
Not graded on evidence
Outcome: Can’t be checked
Both AI providers screened this claim on their own and found that public evidence could not settle it. It is recorded and attributed, not scored.
Why it can’t be checked
Each screen gave its own reason:
- Too vague to test: informed readers would test it in different ways and reach different answers.
- Stated as a chance or a possibility, or at medium or low confidence.
- Evidence
- Not checked
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Not checked against evidence. Two independent screens, one from each provider, both found that evidence could not settle it: one found it too vague to test: informed readers would test it in different ways and reach different answers; the other found it stated as a chance or a possibility, or at medium or low confidence.
Limitations
None recorded.
Evidence (0 items)
No evidence was searched for. Both screens found that public evidence could not settle this claim.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
And of course advancing on benchmarks isn’t itself enough anyway; few benchmarks capture the complexity of the real world. Even if LLMs could max out on all existing benchmark, we might still have a long way to go.
- Where
- paragraph 24
- Published
- 2024-04-13
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-evidence-that-llms-are-reaching-a-eac443-a1: Current. Outcome: Can’t be checked. Not checked against evidence. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-evidence-that-llms-are-reaching-a-eac443-a1.