“Statistical word prediction is no substitute for real math, but more than a few people have thought that it was.”
Our reading, the sentence that was checked:
Statistical word prediction cannot substitute for real mathematical computation.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Scientists, please don’t let your chatbots grow up to be co-authors, paragraph 7.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-scientists-please-dont-let-your-chatbots-5e69a7-a3
We recorded 15 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-01-14. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Statistical word prediction is no substitute for real math
Our interpretation
- In the surrounding discussion of a pace calculation, the ordinary reading is that contemporary unaided ChatGPT was not a dependable replacement for mathematical computation requiring correct numerical answers.
- This is not interpreted as a theorem that all future language models must fail at mathematics, or that contemporary models could solve no mathematical problems.
- For this ongoing observation, use the publicly observable period from ChatGPT’s launch through the post, November 30, 2022 to January 14, 2023.
- The supplied normalized proposition concerns mathematical substitutability; the accompanying assertion about how many people believed otherwise is not separately quantified.
Counts as supported if: Contemporary demonstrations show plausible but wrong mathematical results from unaided ChatGPT on tasks handled correctly by direct computation.
Counts as contradicted if: Contemporary evaluations establish dependable replacement of direct computation across the relevant numerical tasks, and the alleged failures are shown to be misreported or outside that scope.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
On January 9, 2023, Wolfram compared unaided ChatGPT’s mathematical answers with computational results and showed plausible but incorrect answers [e1]. This supports the contextual claim that contemporary unaided ChatGPT was not a dependable substitute for precise numerical computation [e1].
Limitations
- The demonstration provides no representative mathematical error rate and does not establish an impossibility applying to all future language models or systems using computational tools [e1].
- The January 31, 2023 evaluation’s supplied abstract does not establish when responses were collected, so it cannot establish performance during the November 30, 2022–January 14, 2023 target period [e2].
- The November 30, 2022 launch disclosure acknowledges incorrect answers but does not measure mathematical accuracy [e3].
Evidence (3 items)
Supports the claim: Stephen Wolfram,
it often falls down.
Why it matters here: Wolfram presents direct comparisons in which ChatGPT’s plausible mathematical answers fail checks against computational results. These support the narrow claim that unaided contemporary output could not be treated as dependable calculation.
- Publisher
- Stephen Wolfram
- Kind of source
- Demonstration
- Describes events of
- Public from
Context only: Simon Frieder and colleagues / arXiv,
fails to provide correct solutions.
Why it matters here: The later evaluation reports substantial mathematical limitations. It is not decisive for the target period because the retrieved abstract does not establish when the tested responses were collected.
- Publisher
- Simon Frieder and colleagues / arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: OpenAI,
plausible-sounding but incorrect or nonsensical answers.
Why it matters here: The launch disclosure establishes the beginning of the observation period and acknowledges incorrect answers. It does not itself measure mathematical accuracy.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Scientists, please don’t let your chatbots grow up to be co-authors
Statistical word prediction is no substitute for real math, but more than a few people have thought that it was.
- Where
- paragraph 7
- Published
- 2023-01-14
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-scientists-please-dont-let-your-chatbots-5e69a7-a3: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-scientists-please-dont-let-your-chatbots-5e69a7-a3.
HeldBy this site
The final-text check failed: rationale_supported_by_evidence, citations_correct; overclaims: The January 31, 2023 evaluation does not establish when its tested responses were collected, so it cannot establish performance during the November 30, 2022–January 14, 2023 target period [e2]..
About assessment marcus-scientists-please-dont-let-your-chatbots-5e69a7-a2, which was not published.