“failed, multiple times, to properly count to 50”
Our reading, the sentence that was checked:
In Marcus's test, ChatGPT failed several times to count to 50 correctly.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, ChatGPT in Shambles, paragraph 29.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-chatgpt-in-shambles-549d1f-a2
We recorded 6 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-02-04. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- in a 10-minute test session (qualifier, stated by the speaker)
From the speaker
- ChatGPT
- failed, multiple times
- to properly count to 50
- over the course of 10 minutes
- latest (unpaid) version of ChatGPT
Our interpretation
- Read as: in Marcus's own chat session on or about 2025-02-04, more than one ChatGPT answer that was meant to list all 50 US states had a number of rows other than 50.
- The claim is about one specific session, not about ChatGPT in general. The primary record is the shared conversation Marcus linked.
Counts as supported if: The linked conversation, or an independent check of it, shows at least two answers that were meant to cover all 50 states but had a row count other than 50.
Counts as contradicted if: The conversation shows that at most one answer had the wrong number of states, or that the answers were not meant to list all 50.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The claim is about one session with the free version of ChatGPT on or about 4 February 2025. Marcus describes the session as lasting 10 minutes. The contract reads "count to 50" as meaning that answers meant to list all 50 states had a row count other than 50. The claim is supported only if the shared transcript, or an independent check of it, shows at least two such answers. The evidence includes neither the transcript nor such a check. A post by Benjamin Todd, available on 9 February 2025, says "GPT-4o misses out a bunch of states" (e1). It does not say whether Todd looked at Marcus's transcript. It also does not give a row count for any of Marcus's tables or say how many tables were wrong. A reader comment dated 5 February 2025 says the reader's own GPT-4o run left out states (e2). That was a different session. Neither item shows that two or more of Marcus's tables had a count other than 50.
Limitations
- The shared transcript linked in Marcus's post was not found, so the rows in his tables could not be counted.
- e1 does not show whether its author checked Marcus's transcript, and it gives no row counts.
- e2 describes a different session and cannot show what Marcus's session produced.
- The contract notes that "count to 50" is figurative. Its reading as table row counts is an interpretation.
Evidence (2 items)
Context only: Benjamin Todd (Substack),
GPT-4o misses out a bunch of states.
Why it matters here: A critic seems to accept that the free GPT-4o run left out states. It is unclear whether he checked Marcus's transcript or is just restating it, and he does not say how many tables were wrong.
- Publisher
- Benjamin Todd (Substack)
- Kind of source
- Secondary summary
- Describes events of
- Public from
Context only: Substack reader comment,
I just ran the same prompt using my subscription model (4o) and the results are crap. States missing.
Why it matters here: A reader's separate GPT-4o run also left out states. This makes Marcus's account plausible, but it is not his session.
- Publisher
- Substack reader comment
- Kind of source
- Secondary summary
- Describes events of
- Public from
What would settle it
The chatgpt.com conversation shared in Marcus's post, or an independent check of it, with the rows counted in each answer meant to list all 50 states.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, ChatGPT in Shambles
failed, multiple times, to properly count to 50
- Where
- paragraph 29
- Published
- 2025-02-04
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-chatgpt-in-shambles-549d1f-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.