“ChatGPT couldn’t, for example, reliably count to four (figure 1) or do one-digit arithmetic in the context of simple word problem (figure 2).”
Our reading, the sentence that was checked:
ChatGPT could not reliably count to four or do one-digit arithmetic in simple word problems.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Large Language Models like ChatGPT say The Darnedest Things, paragraph 3.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-large-language-models-like-chatgpt-abae3d-a2
We recorded 6 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-01-09. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- couldn’t
- reliably count to four
- do one-digit arithmetic
- in the context of simple word problem
Our interpretation
- The plain reading is inconsistent success on elementary counting and single-digit arithmetic word problems, not an inability ever to answer them correctly.
- The following paragraph explicitly allows correct responses, so isolated successes cannot contradict the claim.
- The two task families must be assessed separately. Errors on algebra, larger numbers or later models cannot settle either family.
- The target period is launch through the post date. I do not introduce a numerical reliability cutoff that the speaker did not supply.
Counts as supported if: Independent repeated trials or a sufficiently varied set of dated launch-era tests show recurring failures both in counting to four and in simple word problems using single-digit arithmetic.
Counts as contradicted if: Representative repeated launch-era tests show consistently correct performance on both task families, with enough coverage to exclude the claimed practical unreliability. If only one family is settled, the full claim remains partly unresolved.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The cited example records an algebra error on January 24, 2023, after the claim's January 9 cutoff, and involves larger quantities [e1]. The paper also describes an early-January experiment, but does not date it precisely enough to establish that it preceded the cutoff or isolate the two claimed task families [e1]. This evidence therefore settles neither reliability in counting to four nor reliability in simple single-digit arithmetic word problems during the launch period [e1].
Limitations
- The packet provides no independent repeated or sufficiently varied launch-period tests for either specified task family.
- A later algebra error does not establish reliability on elementary counting or single-digit arithmetic word problems.
- The claim supplies no numerical reliability threshold.
Evidence (1 item)
Context only: arXiv; Paulo Shakarian and coauthors,
algebraic operation
Why it matters here: The paper records a mathematical error, but the example occurred after the post and involves algebra with larger quantities. It cannot establish reliability on counting to four or single-digit word problems during the target period.
- Where
- Figure 1 caption
- Publisher
- arXiv; Paulo Shakarian and coauthors
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Independent archived tests from November 30, 2022 through January 9, 2023, preserving model versions, dates, prompts, conversational context and outputs, with repeated trials and varied formulations assessed separately for counting to four and simple single-digit arithmetic word problems.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Large Language Models like ChatGPT say The Darnedest Things
ChatGPT couldn’t, for example, reliably count to four (figure 1) or do one-digit arithmetic in the context of simple word problem (figure 2).
- Where
- paragraph 3
- Published
- 2023-01-09
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-large-language-models-like-chatgpt-abae3d-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-large-language-models-like-chatgpt-abae3d-a2.