“Second, GPT-4 doesn’t reliably understand how to use the tools.”
Our reading, the sentence that was checked:
GPT-4 does not reliably understand how to use the Code Interpreter and Wolfram Alpha plug-ins.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Getting GPT to work with external tools is harder than you think, paragraph 4.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-getting-gpt-to-work-with-external-0ab0f2-a1
We recorded 16 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-10-23. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- the Code Interpreter and Wolfram Alpha plug-ins (scope, stated by the speaker)
From the speaker
- GPT-4
- doesn’t reliably understand how to use the tools
- Code Interpreter
- Wolfram Alpha
- A couple of months ago
Our interpretation
- Read 'understand how to use' behaviorally: choosing appropriate tool operations, making useful calls, and correctly handling their results.
- Read 'doesn’t reliably' as recurring consequential failures, not an assertion that every call fails or that GPT-4 has no useful tool skills.
- Anchor the observation to the recent 105-problem testing discussed in the post. The linked study dates that testing to June–August 2023.
Counts as supported if: Contemporaneous records show recurring avoidable failures in selecting, invoking, or interpreting the named tools, including examples involving both plug-ins.
Counts as contradicted if: Comparable testing shows consistently appropriate tool use, with alleged failures attributable to unrelated problem-solving errors or incorrect grading rather than tool use.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The contemporary test records document recurring failures in how GPT-4 selected operations, invoked tools, and handled results. Examples involve both Wolfram Alpha and Code Interpreter. This supports the ordinary behavioral reading of the claim. It does not establish that tool use usually failed, or that GPT-4 lacked useful tool skills.
Limitations
- The main evidence comes from the study underlying the post, rather than an independent replication.
- The study used a small, deliberately constructed collection and published edited outputs.
- No numerical reliability threshold was specified.
Evidence (2 items)
Supports the claim: Ernest Davis and Scott Aaronson, hosted by New York University,
GPT-4 also sometimes creates quite useless calls to the plug-ins.
Why it matters here: By completion of testing, the report documented repeated unsuccessful Wolfram queries, failure to request a usable numerical representation, and Code Interpreter bypasses that produced calculation errors. These distinguish tool-use failures from merely misunderstanding a science problem.
- Publisher
- Ernest Davis and Scott Aaronson, hosted by New York University
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: Ernest Davis and Scott Aaronson, hosted by New York University,
It claimed that the relative error is 0.
Why it matters here: The recorded answer used Monte Carlo sampling for an extremely small nonzero probability, returned zero, and claimed zero relative error. This illustrates inappropriate computational-method selection and interpretation of the result, rather than a malfunction of Python.
- Publisher
- Ernest Davis and Scott Aaronson, hosted by New York University
- Kind of source
- Independent measurement
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Getting GPT to work with external tools is harder than you think
Second, GPT-4 doesn’t reliably understand how to use the tools.
- Where
- paragraph 4
- Published
- 2023-10-23
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-getting-gpt-to-work-with-external-0ab0f2-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-getting-gpt-to-work-with-external-0ab0f2-a1.