“Fall off on increasingly large multiplication problems has repeatedly been observed, both in older models and newer models.”
Our reading, the sentence that was checked:
Both older and newer LLMs have repeatedly been observed to get worse as multiplication problems get larger.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, LLMs don’t do formal reasoning - and that is a HUGE problem, paragraph 13.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-llms-dont-do-formal-reasoning-and-204262-a1
We recorded 9 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-10-11. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Fall off on increasingly large multiplication problems
- has repeatedly been observed
- both in older models and newer models
Our interpretation
- Read as: by October 2024, more than one published evaluation had shown LLM accuracy on multiplication dropping as the number of digits grows, in both older models (for example GPT-3, ChatGPT, GPT-4) and newer ones (for example GPT-4o, o1).
- 'Repeatedly' is read as at least two independent evaluations.
Counts as supported if: Two or more evaluations before the post show multiplication accuracy falling with operand size, covering both older and newer models.
Counts as contradicted if: Evaluations before the post show no fall-off with size for the models tested, or such fall-off was seen only once or only in one generation of models.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Marcus said accuracy falling off on larger multiplication problems had been seen repeatedly, in both older and newer models. For older models, Dziri et al. (2023) found that GPT-3, ChatGPT and GPT-4 went from near-perfect to zero accuracy as problem size grew. ChatGPT and GPT-4 scored only 55% and 59% on 3x3-digit problems. For newer models, Yuntian Deng's September 2024 evaluation went up to 20x20 digits. It found GPT-4o struggles beyond 4x4 (under 30% accuracy, per TechCrunch), and o1 does well only up to about 9x9. That makes at least two separate evaluations across two model generations, all showing the same fall-off. The claim holds as stated.
Limitations
- The Deng results are a researcher's social-media post plus a news report, not a peer-reviewed paper.
- Some models do better at larger sizes than others; the claim is about the trend, not about one threshold.
Evidence (4 items)
Supports the claim: arXiv (Dziri et al., NeurIPS 2023),
task performances deteriorate significantly from near perfection to zero with increasing complexity when measured by either problem size
Why it matters here: For older models (GPT-3, ChatGPT, GPT-4), multi-digit multiplication accuracy falls from near perfect to zero as problem size grows.
- Publisher
- arXiv (Dziri et al., NeurIPS 2023)
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Supports the claim: arXiv (Dziri et al., NeurIPS 2023),
Yet, off-the-shelf ChatGPT and GPT4 achieve only 55% and 59% accuracies on this task, respectively (§3).
Why it matters here: Gives concrete numbers: even at 3x3 digits, ChatGPT and GPT-4 were well short of perfect.
- Publisher
- arXiv (Dziri et al., NeurIPS 2023)
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Supports the claim: Yuntian Deng (University of Waterloo) on X,
We tested it on up to 20x20 multiplication—o1 solves up to 9x9 multiplication with decent accuracy, while gpt-4o struggles beyond 4x4.
Why it matters here: A separate 2024 evaluation of newer models (GPT-4o, o1) finds both hold up only to a certain size and then decline.
- Where
- Post 1/4
- Publisher
- Yuntian Deng (University of Waterloo) on X
- Kind of source
- Independent measurement
- Describes events of
- Public from
Supports the claim: TechCrunch,
“GPT-4o struggles with multi-digit multiplication, achieving less than 30% accuracy beyond four-digit by four-digit problems,” Deng told TechCrunch.
Why it matters here: Gives a numeric threshold for the newer GPT-4o: accuracy falls below 30% past 4x4 digits.
- Publisher
- TechCrunch
- Kind of source
- News report
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, LLMs don’t do formal reasoning - and that is a HUGE problem
Fall off on increasingly large multiplication problems has repeatedly been observed, both in older models and newer models.
- Where
- paragraph 13
- Published
- 2024-10-11
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-llms-dont-do-formal-reasoning-and-204262-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-llms-dont-do-formal-reasoning-and-204262-a1.