“[Aside from this, models like o3 actually hallucinate a bunch more than attentive humans, struggle heavily with drawing reliable diagrams, etc; they happen to share a few weakness with humans, but on a bunch of dimensions they actually fall short.]”
Our reading, the sentence that was checked:
Models like o3 hallucinate much more than attentive humans and struggle with reliable diagrams.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, A knockout blow for LLMs?, paragraph 30.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-a-knockout-blow-for-llms-ebeec8-a2
We recorded 19 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-06-07. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- models like o3
- hallucinate a bunch more than attentive humans
- struggle heavily with drawing reliable diagrams
Our interpretation
- Hallucination means presenting fabricated or unsupported factual content as true.
- The human comparison requires attentive people answering comparable tasks with comparable access to information and an equivalent option to abstain.
- I read 'a bunch more' as a substantial difference, not any detectable difference.
- Reliable diagrams must represent the requested entities, relationships, and constraints correctly; visual attractiveness alone is insufficient.
- The target is the contemporary o3 assessment period from its April 2025 release through the post. This is an ongoing-state claim, not a claim about later models.
Counts as supported if: Matched evaluations show substantially more hallucinations from o3-like models than attentive humans, and relevant diagram-generation tests show substantial correctness failures.
Counts as contradicted if: Comparable evidence establishes no substantial hallucination disadvantage against attentive humans, or consistently correct diagram generation in the relevant task range. Opposing findings for the two clauses would warrant mixed.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
OpenAI's system card establishes that o3 hallucinated on its tests, but not that it hallucinated substantially more than attentive humans. The diagram evidence found also does not adequately match the model, task, and period. Neither the conjunction nor a supported-versus-contradicted split is established.
Limitations
- The claim combines two distinct empirical assertions.
- Human comparison conditions and diagram correctness standards are unspecified.
- The search gaps do not demonstrate that either assertion is false.
Evidence (1 item)
Context only: OpenAI,
leading to more accurate claims as well as more inaccurate/hallucinated claims.
Why it matters here: The system card documents o3 hallucinations and compares its results with o1 and o4-mini. It supplies no matched attentive-human baseline and therefore cannot establish the claimed human disadvantage.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
What would settle it
Matched factual-error tests of contemporary o3 and attentive humans, plus diagram-generation evaluations with specified tasks, model configuration, and correctness scoring.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, A knockout blow for LLMs?
[Aside from this, models like o3 actually hallucinate a bunch more than attentive humans, struggle heavily with drawing reliable diagrams, etc; they happen to share a few weakness with humans, but on a bunch of dimensions they actually fall short.]
- Where
- paragraph 30
- Published
- 2025-06-07
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-a-knockout-blow-for-llms-ebeec8-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.