“Take hallucinations, a deep, serious, unsolved problem with Generative AI, which I have been warning about since 2001.”
Our reading, the sentence that was checked:
Hallucination is a deep, serious, unsolved problem with generative AI.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, AI literacy, hallucinations, and the law: A case study, paragraph 2.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-ai-literacy-hallucinations-and-the-2d9d2c-a2
We recorded 9 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-05-24. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- hallucinations
- deep, serious, unsolved problem
- with Generative AI
Our interpretation
- Read as a claim about the state of generative AI at the post date (May 2025): leading models still hallucinated at material rates, and no developer had shown a fix.
- 'Deep, serious' is a judgment. Plain reading: material hallucination rates on published evaluations from leading labs, plus documented real-world harm. 'Unsolved' means no lab claimed or showed that hallucinations had been eliminated.
- The part about warning since 2001 is biographical and is not scored.
Counts as supported if: Evaluations of frontier models from around April–May 2025 show substantial hallucination rates, and documented incidents of harmful fabrication continued at that time.
Counts as contradicted if: Evaluations from that period show frontier models with negligible hallucination rates, or credible evidence that the problem had been solved by May 2025.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The contract asks whether frontier models had substantial hallucination rates around April–May 2025, and whether harmful fabrication incidents were still happening then. Both parts are met. OpenAI's system card of April 16, 2025 reported PersonQA hallucination rates of 33% for o3 and 48% for o4-mini, against 16% for the older o1. It reported SimpleQA rates of 51% and 79%. OpenAI wrote that more research was needed to understand the cause. On the same date, Transluce reported that o3 often made up actions it had taken and justified them when challenged. It said this behaviour also appeared in o3-mini and o1. On May 22, 2025, an LA Times column reported that Damien Charlotin's list held 99 court cases with AI hallucinations. It said that in some 30 cases involving lawyers, the errors were found or filed as recently as that year. No evidence in the packet shows that the problem had been solved by May 2025. 'Deep' and 'serious' are judgments, but the evidence fits the plain reading.
Limitations
- Hallucination rates depend heavily on the benchmark and the model. The cited figures come from a few evaluations and do not give one general rate.
- 'Deep' and 'serious' have no measurable threshold.
- The court-case counts cover only publicly reported cases and do not measure how often hallucinations happen overall.
Evidence (3 items)
Supports the claim: OpenAI,
More research is needed to understand the cause of these results.
Why it matters here: OpenAI's own system card for its newest models reports PersonQA hallucination rates of 33% (o3) and 48% (o4-mini), higher than o1, and says the cause is not understood. This shows the problem was material and unsolved in April 2025.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Supports the claim: Transluce,
we found that o3 frequently fabricates actions it took to fulfill user requests, and elaborately justifies the fabrications when confronted by the user.
Why it matters here: An independent lab found that a frontier model often made things up in April 2025, and that the behaviour also appeared in other widely deployed models.
- Publisher
- Transluce
- Kind of source
- Independent measurement
- Describes events of
- Public from
Supports the claim: Los Angeles Times (via Yahoo News),
in some 30 cases involving lawyers the AI-generated errors were discovered or were in documents filed as recently as this year
Why it matters here: Documents real-world harm from hallucinations continuing in 2025, which supports 'serious' and 'unsolved'.
- Publisher
- Los Angeles Times (via Yahoo News)
- Kind of source
- News report
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, AI literacy, hallucinations, and the law: A case study
Take hallucinations, a deep, serious, unsolved problem with Generative AI, which I have been warning about since 2001.
- Where
- paragraph 2
- Published
- 2025-05-24
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-ai-literacy-hallucinations-and-the-2d9d2c-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-ai-literacy-hallucinations-and-the-2d9d2c-a2.