“Imagining perfectly reliable, hallucination-free chatbots by this time next year is bonkers.”
Our reading, the sentence that was checked:
Perfectly reliable, hallucination-free chatbots will not exist by February 2025.
Assessment: Supported Deadline passed on 2026-10-29 Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, No, RAG is probably not going to rescue the current situation, paragraph 30.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-no-rag-is-probably-not-going-to-rescue-559937-a2
We recorded 15 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-02-23. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Forecast
- Deadline, as the speaker wrote it
- “by this time next year”
- Deadline state when assessed
- Deadline passed
- Deadline state on 2026-10-29
- Deadline passed (the release’s reference date)
From the speaker
- perfectly reliable
- hallucination-free chatbots
- by this time next year
Our interpretation
- Read the sentence as predicting that perfectly reliable general-purpose chatbots would not be achieved by February 23, 2025.
- The preceding discussion supplies broad query coverage. A narrowly constrained application or zero errors on one finite test does not establish perfect reliability.
- Assess publicly demonstrated chatbot capabilities. Improved accuracy is compatible with this negative forecast.
Counts as supported if: Public chatbot evaluations and release disclosures through the deadline continue to document factual fabrication or other material reliability failures, with no verified general-purpose system meeting the perfect-reliability standard.
Counts as contradicted if: A chatbot available by February 23, 2025 has reproducibly demonstrated hallucination-free, reliable performance across the broad query scope described, rather than only a selected task or dataset.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Did the forecast come true?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- Deadline passed
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
OpenAI's October 30, 2024 SimpleQA disclosure documented factual errors in models tested without browsing [e1]. Its February 2, 2025 deep-research release disclosure acknowledged that the browsing-enabled capability could hallucinate facts [e2]. The December 6, 2024 Acurai preprint reported success on 37 examples, restricted to factual, mostly relevant passages and small contexts [e3]. These results support the forecast under the contract's public-demonstration standard: the packet documents continuing failures before February 23, 2025 and contains no verified demonstration of perfect general-purpose reliability [e1, e2, e3].
Limitations
- The outcome evidence is concentrated on OpenAI systems [e1, e2].
- SimpleQA's tests without browsing do not establish the performance of retrieval-enabled chatbots [e1].
- The packet does not establish universal nonexistence; Acurai's finite, restricted test does not establish universal reliability [e1, e2, e3].
Evidence (3 items)
Supports the claim: OpenAI,
false outputs
Why it matters here: The release documents continuing factual errors and evaluates several OpenAI models. Its tests exclude browsing, so they establish limitations of those model configurations rather than every retrieval-enabled chatbot.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Supports the claim: OpenAI,
hallucinate facts
Why it matters here: OpenAI's February launch disclosure acknowledges factual hallucinations even in its research capability that searches the web. The same paragraph reports improvement over existing ChatGPT models, which is compatible with continued failure to achieve perfection.
- Where
- Limitations
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: Wood and Forbes, arXiv,
three passages.
Why it matters here: The authors report eliminating hallucinations in their tested sample, providing material counterevidence. Their sample of 37 and restrictions to factual passages and small contexts do not establish a perfectly reliable general-purpose chatbot.
- Publisher
- Wood and Forbes, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, No, RAG is probably not going to rescue the current situation
Imagining perfectly reliable, hallucination-free chatbots by this time next year is bonkers.
- Where
- paragraph 30
- Published
- 2024-02-23
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-no-rag-is-probably-not-going-to-rescue-559937-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-no-rag-is-probably-not-going-to-rescue-559937-a2.