“There is no question that the two plug-ins significantly extend GPT-4’s ability to solve math and science problems.”
Our reading, the sentence that was checked:
The Code Interpreter and Wolfram Alpha plug-ins significantly extend GPT-4's ability to solve math and science problems.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Getting GPT to work with external tools is harder than you think, paragraph 17.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-getting-gpt-to-work-with-external-6ad253-a2
We recorded 16 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-10-23. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- the two plug-ins
- significantly extend GPT-4’s ability
- solve math and science problems
- GPT-4 with either plug-in
Our interpretation
- Compare GPT-4 with each plug-in separately against GPT-4 without that plug-in on comparable problems.
- Read 'significantly' as a substantial practical improvement, not a commitment to a particular statistical test.
- Read the claim as extending useful problem-solving capabilities, not improving every problem or every subject.
- Assess the product configurations available around the 2023 post, not later models or replacement tools.
Counts as supported if: Primary comparisons show substantial practical improvements from each named plug-in on matched mathematical problems, with additional evidence that the capabilities help solve science problems.
Counts as contradicted if: Comparable evaluations show no meaningful extension from either or both plug-ins, or show that apparent gains disappear when material differences in the comparisons are accounted for.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The comparison available on October 3, 2023 reports success on the same 120 difficult MATH problems of 48.33% with Code Interpreter and 45.0% with Wolfram Alpha, versus 30.0% for vanilla GPT-4 [e1]. The study available on September 16, 2023 adds science evidence: Code Interpreter improved correct answers from 0/10 to 6/10 on a line-charge problem and from 1/10 to 10/10 on a two-charge force problem, without consensus-answer voting [e2]. These results support substantial practical gains for both plug-ins in math, with additional science benefits demonstrated for Code Interpreter [e1, e2].
Limitations
- Product and hidden model settings were not fully controlled, limiting precise causal attribution [e1, e2].
- The science study tested only 13 selected math and electromagnetism problems and does not establish science benefits for Wolfram Alpha [e2].
- The studies do not establish improvement across all math and science subjects or report exact experiment dates [e1, e2].
Evidence (2 items)
Supports the claim: Qingyun Wu and colleagues, arXiv,
120 randomly selected level-5 problems
Why it matters here: On the matched 120-problem subset, figure 4(a) reports 48.33% success for ChatGPT with Code Interpreter, 45.0% with Wolfram Alpha, and 30.0% for vanilla GPT-4. These are substantial practical gains for both plug-ins, although the comparison does not fully isolate differences between the web products and baseline configuration.
- Publisher
- Qingyun Wu and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: Tanuj Kumar and Mikhail A. Kats, arXiv,
accurate symbolic manipulations and numerical calculations.
Why it matters here: Repeated trials on the same 13 math and electromagnetism problems show clear gains: the line-charge problem improves from 0/10 correct to 6/10, and the two-charge force problem from 1/10 to 10/10. These are individual-trial results, separate from the paper's additional consensus-answer procedure.
- Publisher
- Tanuj Kumar and Mikhail A. Kats, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Getting GPT to work with external tools is harder than you think
There is no question that the two plug-ins significantly extend GPT-4’s ability to solve math and science problems.
- Where
- paragraph 17
- Published
- 2023-10-23
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-getting-gpt-to-work-with-external-6ad253-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-getting-gpt-to-work-with-external-6ad253-a2.