“LLMs still hallucinate and continue to make boneheaded errors.”
Our reading, the sentence that was checked:
As of February 2026, LLMs still hallucinate and still make boneheaded errors.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, BREAKING: LLM “reasoning” continues to be deeply flawed, paragraph 5.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-breaking-llm-reasoning-continues-4f542d-a2
We recorded 4 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2026-02-10. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- LLMs still hallucinate
- continue to make boneheaded errors
Our interpretation
- Plain reading: current frontier LLMs around February 2026 still produce false or unsupported content at measurable rates, and still fail at some tasks that are simple for people.
- 'Boneheaded errors' read as failures on easy or simple problems, not a specific rate.
Counts as supported if: Measurements from late 2025 to early 2026 show current frontier models with non-trivial hallucination rates, and research documents failures on simple tasks.
Counts as contradicted if: Evidence that frontier LLMs in this period had effectively eliminated hallucinations and simple-task errors.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Vectara's hallucination leaderboard, published 19 November 2025, found that notable thinking models still hallucinated in summaries. It reported GPT-5, Claude Sonnet 4.5, GPT-OSS-120B, Grok-4 and DeepSeek-R1 all above 10%, and the just-released Gemini-3-pro at 13.6% (e1, e2). A survey by Song, Han and Goodman, submitted to arXiv on 5 February 2026, says significant reasoning failures persist in LLMs, 'occurring even in seemingly simple scenarios' (e3). That matches the plain reading of 'boneheaded errors.' Nothing in the evidence shows that hallucinations or simple-task errors had been eliminated. Both parts of the claim hold for November 2025 to February 2026.
Limitations
- Vectara measures faithfulness in summaries only; it does not cover other kinds of hallucination.
- The OpenAI system card figures for GPT-5.2 were not read from the primary source.
- 'Boneheaded' has no threshold; the support rests on the survey's statement about failures in simple scenarios, not on a measured rate.
- The survey abstract does not say how often simple-scenario failures occur in frontier models.
Evidence (3 items)
Supports the claim: Vectara,
Other notable thinking models like Claude Sonnet 4.5, GPT-5, GPT-OSS-120B, Grok-4, or Deepseek-R1 all have a hallucination rate > 10%.
Why it matters here: An independent benchmark in November 2025 found that major frontier reasoning models hallucinated in more than 10% of summaries.
- Publisher
- Vectara
- Kind of source
- Independent measurement
- Describes events of
- Public from
Supports the claim: Vectara,
the just-released Gemini-3-pro, which demonstrates top of the line reasoning capabilities, has a 13.6% hallucination rate
Why it matters here: The newest frontier model at the time still hallucinated at a measurable rate.
- Publisher
- Vectara
- Kind of source
- Independent measurement
- Describes events of
- Public from
Supports the claim: arXiv (Song, Han, Goodman),
Despite these advances, significant reasoning failures persist, occurring even in seemingly simple scenarios.
Why it matters here: A February 2026 survey reports that LLMs still fail even in simple scenarios. This matches the plain reading of 'boneheaded errors.'
- Where
- Abstract
- Publisher
- arXiv (Song, Han, Goodman)
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, BREAKING: LLM “reasoning” continues to be deeply flawed
LLMs still hallucinate and continue to make boneheaded errors.
- Where
- paragraph 5
- Published
- 2026-02-10
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-breaking-llm-reasoning-continues-4f542d-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-breaking-llm-reasoning-continues-4f542d-a2.