“And, indeed, much as I anticipated here two weeks ago, the problems I have been pointing out over the last quarter century still lingered.”
Our reading, the sentence that was checked:
GPT-5 still has the longstanding problems Marcus has identified, such as chess rule-following and world models.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it., paragraph 21.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-gpt-5-overdue-overhyped-and-underwhelming-ca66af-a1
We recorded 23 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-08-09. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- the problems I have been pointing out over the last quarter century
- still lingered
- chess and world models
- GPT-5 still struggles with following the rules
Our interpretation
- Read as: GPT-5 at launch (August 2025) still showed the kinds of failures Marcus has long cited, with chess rule-following as the named example. It does not say GPT-5 failed to improve at all.
- Target period is August 2025, the launch month described in the post.
Counts as supported if: Independent tests of launch-era GPT-5 show it still makes illegal chess moves or similar world-model failures.
Counts as contradicted if: Systematic tests of launch-era GPT-5 show it reliably follows chess rules and keeps a consistent board state, with no such failures.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Marcus said GPT-5 still had his long-cited problems, and he named chess rule-following as the example. On August 7, 2025, Mathieu Acher showed that GPT-5 and GPT-5 Thinking both played an illegal move after a short four-move opening, as earlier GPT models had done. GPT-5 Thinking kept the illegal move even after being told it was illegal. That fits the claim that these failures lingered. OpenAI did report fewer hallucinations, so some problems shrank. But the claim says the problems were still there, not that nothing improved.
Limitations
- The main evidence is a single demonstration, not a benchmark.
- Acher himself warns against over-interpreting one chess example.
- The broader 'quarter century' list of problems is not tested here item by item.
Evidence (3 items)
Supports the claim: Mathieu Acher (blog),
Using a simple four-move sequence, I suceed to force GPT-5 and GPT-5 Thinking into an illegal move. Basically as GPT3.5, GPT4, DeepSeek-R1, o4-mini, o3
Why it matters here: An independent researcher showed launch-week GPT-5 and GPT-5 Thinking making an illegal chess move, as earlier models did. This shows the chess rule-following problem persisted.
- Where
- opening paragraph
- Publisher
- Mathieu Acher (blog)
- Kind of source
- Demonstration
- Describes events of
- Public from
Supports the claim: Mathieu Acher (blog),
What is worst is when stating to GPT-5 Thinking that the move is illegal, it does not change its answer.
Why it matters here: GPT-5 Thinking kept the illegal move even after being told it was illegal. This points to a weak model of the board state.
- Publisher
- Mathieu Acher (blog)
- Kind of source
- Demonstration
- Describes events of
- Public from
Context only: OpenAI,
GPT‑5 is significantly less likely to hallucinate than our previous models.
Why it matters here: OpenAI reported real gains in reliability. So some long-cited problems shrank, even if they did not disappear.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
And, indeed, much as I anticipated here two weeks ago, the problems I have been pointing out over the last quarter century still lingered.
- Where
- paragraph 21
- Published
- 2025-08-09
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-gpt-5-overdue-overhyped-and-underwhelming-ca66af-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-gpt-5-overdue-overhyped-and-underwhelming-ca66af-a1.