“current models aren’t literally look up tables; they can generalize to some degree, but not enough.”
Our reading, the sentence that was checked:
Current AI models can generalize to some degree but not enough.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Breaking news: Scaling will never get us to AGI, paragraph 4.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-breaking-news-scaling-will-never-38a35b-a1
We recorded 6 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-04-08. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- current models aren’t literally look up tables
- they can generalize to some degree
- but not enough
Our interpretation
- The surrounding sentences connect 'not enough' to reliable reasoning and planning beyond familiar training examples.
- I therefore read this as some successful transfer to new examples alongside substantial failures under modest task changes, rather than a claim of no generalization.
- I do not impose an unstated AGI benchmark or require perfect performance.
- For current models, I use the latest completed calendar quarter before publication, January through March 2024. Model evaluations have no uniform reporting cycle, so this is an explicit recency convention.
Counts as supported if: Evaluations of current models show successful transfer on some new task variants but substantial, systematic failures on other meaningful reasoning or planning variants.
Counts as contradicted if: Representative evaluations show reliable transfer across unfamiliar reasoning and planning variants, undermining the claimed insufficiency.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Under the contextual reading of partial but unreliable generalization, the claim is supported. GSM-Plus finds successful handling of some variations alongside substantial failures on others across many models and prompting methods. Earlier planning research demonstrates meaningful successes, so the evidence supports a reliability limitation rather than a blanket claim that models cannot reason, plan, or generalize.
Limitations
- The outcome depends on interpreting 'not enough' through the surrounding discussion of reliability.
- The evidence does not establish an AGI threshold or a permanent limitation.
- One benchmark family cannot characterize all current AI models and uses.
Evidence (2 items)
Supports the claim: arXiv; Li and colleagues,
struggling with answering the variations
Why it matters here: The study evaluates 25 LLMs and four prompting techniques. It reports resilience to some numerical and wording changes alongside substantial failures under other mathematical perturbations, supporting the contextual claim of partial but unreliable generalization.
- Publisher
- arXiv; Li and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Silver and colleagues,
a surprisingly powerful generalized planner
Why it matters here: GPT-4 generated planning programs that generalized to other tasks with automated debugging support. This challenges any blanket dismissal of planning generalization, but predates the target quarter and does not establish broad reliability.
- Where
- Abstract
- Publisher
- arXiv; Silver and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Breaking news: Scaling will never get us to AGI
current models aren’t literally look up tables; they can generalize to some degree, but not enough.
- Where
- paragraph 4
- Published
- 2024-04-08
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-breaking-news-scaling-will-never-38a35b-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-breaking-news-scaling-will-never-38a35b-a1.