“If two graphs I plotted are remotely correct, Mollick’s claim that “the rate of improvement in Large Language models show capabilities doubling every 5 to 14 months” is no longer holding true.”
Our reading, the sentence that was checked:
If his graphs are roughly correct, LLM capabilities are no longer doubling every 5 to 14 months.
Assessment: Can’t be checked Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Evidence that LLMs are reaching a point of diminishing returns - and what that might mean, paragraph 16.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-evidence-that-llms-are-reaching-a-c5116b-a1
We recorded 20 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-04-13. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- if his two graphs are remotely correct (qualifier, stated by the speaker)
From the speaker
- If two graphs I plotted are remotely correct
- capabilities doubling every 5 to 14 months
- is no longer holding true
Our interpretation
- The intended observation concerns the approximately thirteen months preceding April 13, 2024, not subsequent model development.
- The post does not define a numerical scale of overall capability on which doubling can be tested.
- Doubling raw accuracy, halving errors, and halving compute required for fixed performance are materially different tests. Choosing one would supply the central meaning of the claim.
- The accuracy of the graphs is an evidential premise, not an antecedent to a future forecast.
This claim is not graded on evidence. The record checks that the words are quoted correctly and attributed to the right person, and that the claim is classified correctly.
Current assessment
Not graded on evidence
Outcome: Can’t be checked
The statement is a recommendation, a value judgment, or too open to test. It is recorded, not scored.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The narrower benchmark comparisons are checkable, but this claim makes an undefined quantitative statement about overall capability. The underlying study measures compute efficiency at fixed performance, a different quantity. Selecting accuracy doubling, error halving, or efficiency gains would determine the test rather than merely clarify it. Neither support nor contradiction is fair without that missing definition.
Limitations
- This does not endorse Mollick's wording.
- It does not negate the separately checkable observation of smaller reported MMLU gains.
- The conditional premise does not supply the missing capability measure.
Evidence (1 item)
No outcome evidence: a claim of this kind is recorded and attributed, not scored.
What would settle it
A contemporaneous definition of capability and its doubling rule, together with dated measurements and a specified connection to the two graphs.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
If two graphs I plotted are remotely correct, Mollick’s claim that “the rate of improvement in Large Language models show capabilities doubling every 5 to 14 months” is no longer holding true.
- Where
- paragraph 16
- Published
- 2024-04-13
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-evidence-that-llms-are-reaching-a-c5116b-a1: Current. Outcome: Can’t be checked. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-evidence-that-llms-are-reaching-a-c5116b-a1.