“But, crucially, it often gets these wrong, particularly when they involve spatial reasoning beyond a cookie-cutter formula or reasoning about sequences of disparate events.”
Our reading, the sentence that was checked:
GPT-4 with either plug-in often gets wrong problems that a human solver would need significant thought to translate into math notation, particularly those involving spatial reasoning beyond a cookie-cutter formula or reasoning about sequences of disparate events.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Getting GPT to work with external tools is harder than you think, paragraph 17.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-getting-gpt-to-work-with-external-0a2884-a2
We recorded 14 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-10-23. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- problems a human would need significant thought to translate into math notation (scope, stated by the speaker)
- particularly spatial reasoning beyond a cookie-cutter formula or sequences of disparate events (qualifier, stated by the speaker)
From the speaker
- often gets these wrong
- spatial reasoning beyond a cookie-cutter formula
- reasoning about sequences of disparate events
Our interpretation
- These refers to problems requiring substantial work to translate into mathematics.
- Often means recurring failures across relevant examples, rather than an isolated mistake; it does not necessarily mean most problems.
- The contextual reference to earlier experiments identifies the linked June–August 2023 study as the observation period.
- The assessment concerns the described problem collection. It does not establish a population-wide error rate.
Counts as supported if: The contemporary records show repeated errors by both plug-in-enabled systems on spatial and multi-stage problems requiring nontrivial translation into mathematics.
Counts as contradicted if: The relevant records show consistently correct solutions, with only isolated errors on those problem types.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
In the June–August 2023 tests, both GPT4+WA and GPT4+CI failed the pendulum and chemical-reaction problems [e1]. On the eight eclipse questions, GPT4+WA gave six wrong answers and GPT4+CI gave five; these were marked wrong, rather than partial credit or no answer [e1]. These records support recurring failures on spatial and multi-stage problems within the described collection [e1].
Limitations
- The selected problems do not establish a general error rate for translation-intensive tasks or a measured comparison with human difficulty [e1].
- The scores establish answer failures, but do not establish whether a plug-in was called on each eclipse question [e1].
Evidence (1 item)
Supports the claim: New York University; Ernest Davis and Scott Aaronson,
combining several calculations of different kinds.
Why it matters here: Both systems failed the pendulum and chemical-reaction problems. The spatial eclipse subset contained six wrong answers out of eight for GPT4+WA and five out of eight for GPT4+CI. These are repeated failures in the described collection.
- Publisher
- New York University; Ernest Davis and Scott Aaronson
- Kind of source
- Preprint, not peer reviewed
- Arithmetic
- Wrong answers = 8 minus the sum of the eight scores: WA = 6; CI = 5. (inputs: Table 5 eclipse scores for GPT4+WA: 0, 1, 0, 0, 0, 0, 0, 1.; Table 5 eclipse scores for GPT4+CI: 0, 1, 1, 0, 0, 0, 0, 1.; Each zero in this subset is marked Wrong.; Exact integer counts.)
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Getting GPT to work with external tools is harder than you think
But, crucially, it often gets these wrong, particularly when they involve spatial reasoning beyond a cookie-cutter formula or reasoning about sequences of disparate events.
- Where
- paragraph 17
- Published
- 2023-10-23
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-getting-gpt-to-work-with-external-0a2884-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-getting-gpt-to-work-with-external-0a2884-a2.