“Why LLMs just can’t stop hallucinating.”
Our reading, the sentence that was checked:
LLMs are unable to stop hallucinating.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, AI Cartoon of The Year — and five (re)readings for 2026, paragraph 7.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-ai-cartoon-of-the-year-and-five-rereadings-15925b-a2
We recorded 7 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-12-30. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- LLMs
- just can't stop hallucinating
- listed as an essay title from 2025 recommended for rereading
Our interpretation
- Plain reading: as of the post date, current LLMs, including frontier models, still produce hallucinations (plausible but false content) at measurable, non-trivial rates, and no model has eliminated them.
- The phrase 'can't stop' also hints at an inherent, permanent limit. That stronger theoretical point cannot be settled by observation. This assessment checks only the observable part: whether hallucination persisted in state-of-the-art models in late 2025.
Counts as supported if: Independent benchmarks or developer research from late 2025 show that leading LLMs still hallucinate at non-trivial rates.
Counts as contradicted if: Credible evidence from late 2025 shows that frontier LLMs had reliably eliminated hallucinations, with near-zero rates across standard factuality and faithfulness tests.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Read plainly, the claim says LLMs were still hallucinating as of late 2025. Vectara's leaderboard, published on 19 November 2025, tested models on how faithfully they summarize. It reported that Claude Sonnet 4.5, GPT-5, GPT-OSS-120B, Grok-4 and DeepSeek-R1 each had a hallucination rate above 10%. The lowest rate on that benchmark was 3.3%, for Gemini-2.5-flash-lite, so no model it tested reached zero. In a paper posted on 4 September 2025, OpenAI researchers wrote that hallucinations persist even in state-of-the-art systems. Together these support the observable part of the claim for the September to December 2025 period. The stronger reading, that LLMs can never stop hallucinating, is not tested here. The same OpenAI paper argues that hallucinations could be reduced by changing how benchmarks are scored.
Limitations
- The Vectara figures come from one benchmark of summarization faithfulness. They may not reflect other tasks or other ways of measuring factual accuracy.
- The evidence covers only the models these two sources name or describe. It does not show results for every frontier model released in late 2025.
- The GPT-5.2 system card (December 2025) could not be read, so its hallucination figures are not included.
- This grade covers whether hallucination persisted in late 2025. It does not cover the claim that hallucination can never be eliminated.
Evidence (4 items)
Supports the claim: Vectara,
Other notable thinking models like Claude Sonnet 4.5, GPT-5, GPT-OSS-120B, Grok-4, or Deepseek-R1 all have a hallucination rate > 10%.
Why it matters here: On a summarization faithfulness benchmark, leading frontier models from several labs hallucinated in over 10% of summaries in November 2025.
- Publisher
- Vectara
- Kind of source
- Benchmark result
- Describes events of
- Public from
Supports the claim: Vectara,
On the flipside, Gemini-2.5-flash-lite leads with 3.3% hallucination rate
Why it matters here: Even the best model on this benchmark had a nonzero hallucination rate (3.3%), so no tested model had stopped hallucinating.
- Publisher
- Vectara
- Kind of source
- Benchmark result
- Describes events of
- Public from
Supports the claim: arXiv (OpenAI authors),
Such "hallucinations" persist even in state-of-the-art systems and undermine trust.
Why it matters here: Researchers at a leading LLM developer state that hallucinations persist in state-of-the-art systems. The paper also argues that changing how models are evaluated could reduce them, which bears on the stronger 'inherent' reading.
- Where
- Abstract
- Publisher
- arXiv (OpenAI authors)
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: The Register,
Deloitte has agreed to refund part of an Australian government contract after admitting it used generative AI to produce a report riddled with fake citations, phantom footnotes, and even a made-up quote from a Federal Court judgment.
Why it matters here: A real-world hallucination incident with consequences in late 2025. It is context only, because the model was GPT-4o and the report dates from July 2025.
- Where
- paragraph 1
- Publisher
- The Register
- Kind of source
- News report
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, AI Cartoon of The Year — and five (re)readings for 2026
Why LLMs just can’t stop hallucinating.
- Where
- paragraph 7
- Published
- 2025-12-30
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-ai-cartoon-of-the-year-and-five-rereadings-15925b-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-ai-cartoon-of-the-year-and-five-rereadings-15925b-a2.