“That is in part because you can’t build reliable agents unless you can solve the hallucination problem, and there is still no principled trustworthy answer to that problem.”
Our reading, the sentence that was checked:
As of January 2025 there is no principled, trustworthy solution to the hallucination problem.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, AI Agents: Hype versus Reality, redux, paragraph 11.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-ai-agents-hype-versus-reality-redux-65244e-2-a3
We recorded 6 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-01-15. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- there is still no principled trustworthy answer
- the hallucination problem
- you can't build reliable agents unless you can solve the hallucination problem
Our interpretation
- Only the factual part is scored: that as of January 2025 no method existed that reliably and verifiably eliminated or solved hallucinations in generative AI / LLMs. The claim that reliable agents require solving hallucination is a premise about what agents need and is not scored.
- Plain reading of 'principled trustworthy answer': a method, grounded in theory, that reliably prevents or removes hallucinations across general use, so that outputs can be trusted. Methods that only reduce hallucination rates, or detect some of them, do not count as such an answer.
- Target period: January 2025, the month of the post. The post describes an ongoing state ('still no') without naming a period.
Counts as supported if: Expert or primary sources from around January 2025 say hallucinations had not been eliminated or reliably solved, or frontier models of that time still showed large measured hallucination rates.
Counts as contradicted if: A method existed by January 2025 that was widely accepted, or independently shown, to reliably eliminate hallucinations in general-purpose LLMs.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
On 15 January 2025, Marcus wrote that there was still no principled, trustworthy answer to AI hallucination. Only this factual part is scored. A Nature news feature published 21 January 2025 was headlined 'AI hallucinations can't be stopped — but these techniques can limit their damage'. Its standfirst said large language models were still struggling to tell the truth. A Georgia Tech summary of that feature said researchers were working to make hallucinations less frequent and less problematic. OpenAI's system card for o3-mini, a model dated 31 January 2025, reported a 14.8% hallucination rate on PersonQA. OpenAI wrote that more work was needed to understand hallucinations. A June 2024 Nature paper describes a statistics-grounded method that detects a subset of hallucinations, called confabulations. Detecting a subset does not meet the contract's test of a method that reliably prevents hallucinations in general use. None of the cited sources describes such a method.
Limitations
- 'Principled trustworthy answer' has no stated threshold. The assessment uses the contract's reading: a method that reliably prevents hallucinations in general use.
- The body of the Nature feature was paywalled. The assessment relies on its headline, standfirst and a university summary.
- No January 2025 snapshot of a hallucination leaderboard was found.
- The semantic entropy paper dates from June 2024, before the target period, and is used only as context.
- The view that reliable agents need hallucination solved is a premise and was not assessed.
Evidence (4 items)
Supports the claim: Nature,
Developers have tricks to stop artificial intelligence from making things up, but large language models are still struggling to tell the truth, the whole truth and nothing but the truth.
Why it matters here: A major science journal's survey of the field in January 2025 says hallucinations can't be stopped and that existing techniques only limit the damage. That matches 'no principled trustworthy answer'.
- Publisher
- Nature
- Kind of source
- News report
- Describes events of
- Public from
Supports the claim: OpenAI,
More work is needed to understand hallucinations holistically, particularly in domains not covered by our evaluations (e.g., chemistry).
Why it matters here: OpenAI's newest model at the end of January 2025 still hallucinated on 14.8% of PersonQA questions, and OpenAI said more work was needed to understand hallucinations. No general solution was in hand.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Supports the claim: Georgia Institute of Technology, College of Computing,
Vempala and others are working on ways to make hallucinations less frequent and less problematic.
Why it matters here: Summary of the Nature feature. It shows that researchers' goal in January 2025 was to reduce hallucinations, not that they had solved them.
- Where
- paragraph 2
- Publisher
- Georgia Institute of Technology, College of Computing
- Kind of source
- Secondary summary
- Describes events of
- Public from
Context only: Nature (Farquhar et al.),
Here we develop new methods grounded in statistics, proposing entropy-based uncertainty estimators for LLMs to detect a subset of hallucinations—confabulations—which are arbitrary and incorrect generations.
Why it matters here: The strongest counterpoint I found: a principled, theory-grounded method from 2024. But it only detects a subset of hallucinations and does not prevent them, so it is not a full trustworthy answer.
- Where
- abstract
- Publisher
- Nature (Farquhar et al.)
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, AI Agents: Hype versus Reality, redux
That is in part because you can’t build reliable agents unless you can solve the hallucination problem, and there is still no principled trustworthy answer to that problem.
- Where
- paragraph 11
- Published
- 2025-01-15
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-ai-agents-hype-versus-reality-redux-65244e-2-a3: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-ai-agents-hype-versus-reality-redux-65244e-2-a3.