“GPT-4 and similar current language-based AI programs notoriously struggle with mathematical calculations.”
Our reading, the sentence that was checked:
GPT-4 and similar current language-based AI programs struggle with mathematical calculations.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Getting GPT to work with external tools is harder than you think, paragraph 1.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-getting-gpt-to-work-with-external-c70b85-a1
We recorded 14 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-10-23. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- GPT-4 and similar current language-based AI programs
- struggle with mathematical calculations
Our interpretation
- The plain reading is that contemporary general-purpose language models frequently make calculation errors without external tools, not that they fail every calculation.
- Use the 2023 model generation as the reporting period, restricted to evaluations conducted before the October 23 post. The post specifies no reporting interval, and these systems have no regular reporting quarter.
- Specialized arithmetic models are relevant qualifications but do not represent the general-purpose systems discussed.
Counts as supported if: Contemporary evaluations show substantial calculation error rates for GPT-4 and comparable general-purpose language models.
Counts as contradicted if: Contemporary evaluations show consistently accurate unaided calculations across the numerical operations and operand sizes discussed.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Contemporary testing found substantial arithmetic errors in both GPT-4 and ChatGPT. That supports the claim about general-purpose systems operating without tools. The specialized MathGLM result prevents extending this conclusion to all language models or treating the weakness as an architectural impossibility.
Limitations
- Results depend on operand sizes, operation mixtures, and prompting.
- The paper reports the testing year but does not establish an exact testing day.
Evidence (2 items)
Supports the claim: arXiv; Zhen Yang and colleagues,
In contrast, GPT-4 only manages a meager 18.84% accuracy on the same dataset.
Why it matters here: Table 3 reports exact-answer accuracy of 18.84% for GPT-4 and 10.00% for ChatGPT on 9,592 arithmetic cases. These substantial error rates support the ordinary reading of calculation difficulty.
- Publisher
- arXiv; Zhen Yang and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Zhen Yang and colleagues,
Among the different model scales, MathGLM consistently achieves superior performance.
Why it matters here: The specialized MathGLM-2B model achieves 93.03% exact-answer accuracy on the same mixed-arithmetic dataset. This limits any broader inference that language-model architecture inherently prevents accurate calculation.
- Publisher
- arXiv; Zhen Yang and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Getting GPT to work with external tools is harder than you think
GPT-4 and similar current language-based AI programs notoriously struggle with mathematical calculations.
- Where
- paragraph 1
- Published
- 2023-10-23
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-getting-gpt-to-work-with-external-c70b85-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-getting-gpt-to-work-with-external-c70b85-a1.