“Scale isn’t everything, and hasn’t and won’t be enough to solve any of the problems I just listed above.”
Our reading, the sentence that was checked:
Scaling has not been enough to solve the problems of understanding, compositionality, negation, and reliability in LLMs.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, There Must be Some Misunderstanding, paragraph 10.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-there-must-be-some-misunderstanding-8e10c2-a1
We recorded 7 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-02-08. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Scale isn’t everything
- hasn’t
- enough to solve any of the problems I just listed above
- understand their inputs
- Compositionality
- negation
- Reliability
Our interpretation
- Assess the historical 'hasn't' component separately from the forecast.
- The ordinary historical reading is that scaling achieved by publication had not eliminated these behavioral limitations. It does not say that scaling produced no improvements.
- Understanding is operationalized as correctly interpreting inputs and generalizing their meaning to new combinations; this does not adjudicate a philosophical notion of deep understanding.
- Scale means model size, training data and training compute. Failures in scaled models establish remaining limitations, not an isolated causal effect of scale.
- Use the latest completed calendar year and the publication-year period through the post as the reporting window for contemporary model evaluations.
Counts as supported if: Models produced by the scaling approach still exhibit material failures in behavioral input understanding, composition, negation and reliability during the target period.
Counts as contradicted if: Public evaluations establish that scaling had already broadly resolved at least one listed problem, rather than merely improving selected benchmark scores.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
On the historical, behavioral reading, scaled models still had reliability, composition, negation and generalization problems. The evidence also records improvements, which are compatible with problems remaining unsolved. This supports 'hasn't' as a statement about achieved results. It does not prove that scale caused the failures or could never overcome them.
Limitations
- The result depends on treating understanding as observable interpretation and generalization.
- These studies do not isolate model size, data and compute from all other changes.
- The conclusion concerns the contemporary evaluation record, not every model deployment on the post date.
Evidence (3 items)
Supports the claim: OpenAI,
it still is not fully reliable
Why it matters here: OpenAI describes GPT-4 as a milestone in scaling deep learning while acknowledging factual and reasoning errors. The same disclosure reports reduced hallucinations and improved factuality, showing progress without complete resolution.
- Where
- Limitations
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Supports the claim: arXiv; Nouha Dziri and colleagues,
this performance boost diminishes to zero as complexity increases.
Why it matters here: The study finds failures on compositional tasks in contemporary models, including GPT-4, and poor generalization after extensive task training. Prompting helps some cases, but does not establish dependable composition or generalization.
- Publisher
- arXiv; Nouha Dziri and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: arXiv; Iker García-Ferrero and colleagues,
does not facilitate any successful generalization across all other patterns.
Why it matters here: The study documents persistent negation limitations in open LLMs and failures to transfer learned negation patterns. It supports the behavioral understanding component, although it does not test GPT-4 or isolate scaling as the sole cause.
- Publisher
- arXiv; Iker García-Ferrero and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, There Must be Some Misunderstanding
Scale isn’t everything, and hasn’t and won’t be enough to solve any of the problems I just listed above.
- Where
- paragraph 10
- Published
- 2024-02-08
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-there-must-be-some-misunderstanding-8e10c2-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-there-must-be-some-misunderstanding-8e10c2-a1.