“First, neither GPT-4 nor the plug-ins have the commonsense world knowledge and reasonable abilities needed to reliably translate a word problem into a mathematical calculation.”
Our reading, the sentence that was checked:
Neither GPT-4 nor its plug-ins have the commonsense knowledge and reasoning needed to reliably translate word problems into calculations.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Getting GPT to work with external tools is harder than you think, paragraph 4.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-getting-gpt-to-work-with-external-c5d27d-a1
We recorded 14 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-10-23. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- neither GPT-4 nor the plug-ins
- commonsense world knowledge and reasonable abilities
- reliably translate a word problem into a mathematical calculation
Our interpretation
- Read reasonable abilities as reasoning abilities in context.
- The plain reading concerns recurring failures to form the correct mathematical representation of ordinary situations, not a total absence of knowledge or reasoning.
- Observable solution traces can establish formulation failures, but incorrect final answers alone cannot establish their cause.
- Use the 2023 model generation as the reporting period, restricted to evaluations conducted before October 23. No regular reporting interval is specified.
Counts as supported if: Independent solution traces show recurring commonsense or situation-modeling errors before calculation, across GPT-4 and the named plug-in configurations.
Counts as contradicted if: Representative evaluations show dependable translation into correct calculations, with remaining failures attributable to arithmetic, execution, or output handling instead.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
An independent trace confirms that GPT-4 with Code Interpreter can misrepresent a simple spatial relationship before calculation. The same trace also shows correction after prompting. This supports the existence of formulation errors, but does not establish the claim about dependable translation across GPT-4 and both plug-ins or identify missing commonsense knowledge as their general explanation.
Limitations
- The cited example comes from a college electromagnetism task.
- It does not test Wolfram Alpha or independently diagnose internal knowledge.
- The most directly relevant study was written by a coauthor of the post and is excluded.
Evidence (1 item)
Context only: arXiv; Tanuj Kumar and Mikhail A. Kats,
In this figure, the first response from ChatGPT-4/CI (zero-shot) is shown.
Why it matters here: The trace shows Code Interpreter's GPT-4 failing to recognize that the distance from a hemisphere's center to its surface equals its radius, then correcting the mistake when questioned. This illustrates a formulation failure but cannot establish a general deficit across GPT-4 and both plug-ins.
- Publisher
- arXiv; Tanuj Kumar and Mikhail A. Kats
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Independent contemporary traces from representative word problems, with separate scoring of correct mathematical formulation for unaided GPT-4, Code Interpreter, and Wolfram Alpha.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Getting GPT to work with external tools is harder than you think
First, neither GPT-4 nor the plug-ins have the commonsense world knowledge and reasonable abilities needed to reliably translate a word problem into a mathematical calculation.
- Where
- paragraph 4
- Published
- 2023-10-23
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-getting-gpt-to-work-with-external-c5d27d-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-getting-gpt-to-work-with-external-c5d27d-a1.