“Minerva, for example, is trained on an immense amount of data, and ostensibly captures two-digit arithmetic, but it fails altogether on 4-digit multiplication,”
Our reading, the sentence that was checked:
Minerva, despite immense training data and apparent two-digit arithmetic ability, fails altogether on 4-digit multiplication.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Caricaturing Noam Chomsky, paragraph 43.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-caricaturing-noam-chomsky-016ae2-a1
We recorded 5 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-03-12. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Minerva
- trained on an immense amount of data
- ostensibly captures two-digit arithmetic
- fails altogether on 4-digit multiplication
Our interpretation
- Plain reading: in published evaluations, Minerva does well on two-digit arithmetic but gets near-zero accuracy when multiplying two 4-digit numbers.
- 'Fails altogether' means accuracy at or near zero, not just lower accuracy.
- Target period: from Minerva's public evaluation (June 2022) to the post date.
Counts as supported if: A published evaluation shows Minerva's accuracy on 4-digit by 4-digit multiplication at or near zero, while two-digit arithmetic is high.
Counts as contradicted if: A published evaluation shows Minerva gets a clearly non-trivial share of 4-digit multiplications right (for example well above 10%).
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Minerva's paper (June 2022) did test multiplication by number of digits, using 500 random questions per setting (Appendix H, Figure 8). But the results appear only as a plotted curve, with accuracy against digits in the answer. No sentence in the paper gives a 4-digit multiplication figure. A 2023 paper summarizing that figure says Minerva cannot do any multiplication 'of numbers with more than seven digits'. A 4x4-digit product has 7 or 8 digits, so this fits near-total failure. But the wording is ambiguous and does not cover 7-digit answers. The claim is plausible and broadly in line with the evidence, but no quotable source states that Minerva fails altogether on 4-digit multiplication. So I cannot score it supported or contradicted.
Limitations
- The decisive data exist only in a figure that the text tools cannot read.
- The secondary summary may have read the figure differently from the speaker.
Evidence (2 items)
Context only: arXiv (Lewkowycz et al., Google),
For each operation, and for each choice of number of digits, we sample 500 arithmetic questions at random.
Why it matters here: Confirms Minerva's multiplication accuracy was measured by number of digits (Figure 8, x-axis 'Average number of digits in solution'). The plotted values are not in the text, so the 4-digit result cannot be quoted.
- Publisher
- arXiv (Lewkowycz et al., Google)
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv (FERMAT, Sivakumar & Moosavi),
Lewkowycz et al. (2022) conducted a digit study and highlighted that Minerva is unable to per- form any multiplication of numbers with more than seven digits.
Why it matters here: A secondary summary of Minerva's Figure 8. It puts the failure point at 'more than seven digits', probably digits in the answer. A 4x4-digit product has 7 or 8 digits, so this is consistent with near-total failure on 4-digit multiplication, but it does not state it.
- Publisher
- arXiv (FERMAT, Sivakumar & Moosavi)
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
The numbers behind Minerva's Figure 8 (for example from the supplementary data or the authors), showing multiplication accuracy for 7- and 8-digit answers or 4x4-digit operands.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Caricaturing Noam Chomsky
Minerva, for example, is trained on an immense amount of data, and ostensibly captures two-digit arithmetic, but it fails altogether on 4-digit multiplication,
- Where
- paragraph 43
- Published
- 2023-03-12
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-caricaturing-noam-chomsky-016ae2-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-caricaturing-noam-chomsky-016ae2-a1.