“But more than three years on, it’s still—even after a new update a few days ago focused around math—having lot of problems with easy word problems like this:”
Our reading, the sentence that was checked:
Even after its January 30, 2023 math-focused update, ChatGPT still has many problems with easy word problems.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Gaslighting and reality in AI, paragraph 29.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-gaslighting-and-reality-in-ai-9ae28b-a1
We recorded 4 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-02-02. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- even after the January 30 math update (qualifier, stated by the speaker)
From the speaker
- more than three years on
- even after a new update a few days ago focused around math
- having lot of problems with easy word problems
Our interpretation
- The target is ChatGPT following its January 30, 2023 update.
- The observation describes January 30 through February 2, 2023, from the named update to publication of the post.
- The plain reading is a recurring difficulty with elementary word problems involving quantities and ordinary events. It does not imply failure on most problems or any particular numerical failure rate.
- The surrounding explanation about world models is not part of this assessment.
Counts as supported if: Dated tests or inspectable outputs from the target period show repeated substantive failures across elementary word problems, establishing a recurring weakness after the update.
Counts as contradicted if: Relevant, sufficiently broad testing of the updated version shows consistently reliable performance on elementary word problems, and the alleged failures are shown to be invalid or exceptional.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The January 30 update is confirmed. Davis reported a failed repeat test on February 1, but the retrieved text does not expose the problem or ChatGPT's answer. The formal word-problem study documents errors in tests outside the target period. These records make the claim plausible but do not establish recurring failures on easy word problems during January 30–February 2. They also do not contradict it.
Limitations
- Missing image contents prevent verification of the examples Marcus cited.
- Selected failures alone would not establish a population-wide failure rate, although Marcus supplied no numerical rate.
- Later tests cannot substitute for evidence about the stated historical period.
Evidence (3 items)
Supports the claim: OpenAI,
improved factuality and mathematical capabilities.
Why it matters here: Confirms the update referenced in the claim. It does not establish the resulting accuracy on word problems.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: Ernest Davis,
got this even worse result.
Why it matters here: Davis reports an unsuccessful repeat test within the target period. The retrieved text lacks the prompt and answer, preventing independent verification of the difficulty and alleged error.
- Publisher
- Ernest Davis
- Kind of source
- Demonstration
- Describes events of
- Public from
Context only: Paulo Shakarian, Abhinav Koyyalamudi, Noel Ngu and Lakshmivihari Mareedu / arXiv,
probability of failure
Why it matters here: The mid-February experiment reports substantial word-problem errors and strong sensitivity to whether working is shown. Those tests occurred after the target period and cannot establish the January 30–February 2 outcome.
- Publisher
- Paulo Shakarian, Abhinav Koyyalamudi, Noel Ngu and Lakshmivihari Mareedu / arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Recover dated prompts and complete outputs from the January 30 version during January 30–February 2, including Davis's examples and enough additional elementary word problems to assess whether failures recurred. Independently check answers and preserve the testing method.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Gaslighting and reality in AI
But more than three years on, it’s still—even after a new update a few days ago focused around math—having lot of problems with easy word problems like this:
- Where
- paragraph 29
- Published
- 2023-02-02
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-gaslighting-and-reality-in-ai-9ae28b-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-gaslighting-and-reality-in-ai-9ae28b-a1.