“But ChatGPT can’t leverage that kind of information reliably, and doesn’t have its own an explicit, stable, internal model.”
Our reading:
ChatGPT cannot reliably use factual information like Wikipedia's and has no explicit, stable internal world model.
Assessment: Unresolved
What happened
Was it true for the period described?
Outcome: Unresolved
We checked, and the evidence so far doesn’t settle it.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
OpenAI's prepublication tests support a meaningful factual-reliability concern in models available through ChatGPT. They do not reproduce the claimed cross-context nationality inconsistency, and factual errors alone cannot establish the absence of an internal model. The architectural component remains unsettled, so the combined claim cannot receive a supported verdict.
Caveats
- PersonQA is a targeted factuality evaluation, not an estimate of error rates across all ChatGPT use.
- The evidence does not identify the configuration used in Marcus's anecdote.
Evidence (2 items)
Supports the claim: OpenAI,
The o4-mini model underperforms o1 and o3 on our PersonQA evaluation.
Why it matters: PersonQA tests publicly available facts about people. Reported hallucination rates were 33% for o3 and 48% for o4-mini, supporting the factual-reliability concern in these tested configurations; the evaluation does not establish absence of internal representations.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- About events on
- Published
Context only: OpenAI,
Free users can try o4-mini
Why it matters: The launch announcement establishes that the evaluated model was available in ChatGPT before Marcus's post. It does not establish which model produced his example.
- Where
- Access section
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- About events on
- Published
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Where we had to interpret
- The plain reading has two components: unreliable use of well-documented factual information and absence of an explicit, stable internal representation.
- Reliability means ordinarily dependable factual answers across reasonable variations in context. It does not mean that every answer must be correct.
- The reference to Wikipedia concerns information available for learning; it does not establish that a particular response used browsing or retrieved the relevant page.
- The target is ChatGPT configurations relevant at publication, assessed using the latest applicable public evaluations. Earlier tests are eligible when their configuration remains relevant; there is no June-only boundary.
Right if: Applicable tests show recurring factual errors or contradictions across reasonable contexts, and direct architectural or representation evidence establishes the absence of the claimed stable internal model.
Wrong if: Applicable tests demonstrate dependable factual use across contexts, or direct evidence establishes the explicit, stable internal representation whose existence is denied. Partial findings must be reported separately.
We wrote this test after the outcome was known.
What would settle it
Matched factual-consistency tests of the relevant ChatGPT configuration, together with direct architectural or causal representation evidence addressing the claimed stable internal model.
We found 23 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, Generative AI’s crippling and widespread failure to induce robust models of the world, paragraph 30.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-28. The post may have been edited after 2025-06-28.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, Generative AI’s crippling and widespread failure to induce robust models of the world
But ChatGPT can’t leverage that kind of information reliably, and doesn’t have its own an explicit, stable, internal model.
- Where
- paragraph 30
- Published
- 2025-06-28
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Unresolved. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
A reviewer said our test wasn’t fair to what was actually said.