“Perhaps the only use case that really seemed compelling to either of us economically was search (e.g., using Bing powered by ChatGPT instead of Google Search) — but the technical problems there are immense; there is no reason to think that the hallucination problem will be solved soon.”
Our reading, the sentence that was checked:
There is no reason to think that the hallucination problem, part of the immense technical problems facing the search use case, will be solved soon.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, What if Generative AI turned out to be a Dud?, paragraph 8.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-what-if-generative-ai-turned-out-07e6c8-a2
We recorded 23 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2023-08-13. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- in the context of search, perhaps the only economically compelling use case (scope, stated by the speaker)
From the speaker
- search
- the technical problems there are immense
- no reason to think
- the hallucination problem will be solved soon
Our interpretation
- Assess the grounds for expecting a solution that existed when Marcus wrote, rather than whether hallucinations subsequently disappeared.
- The ordinary reading of a solution is dependable factual answers across ordinary search queries. Improvements on selected tasks alone do not establish that.
- Research has no fixed reporting cycle. Use the latest completed day before publication as the snapshot of publicly available knowledge; earlier experiments are contextual evidence.
- Neither a numerical error tolerance nor a duration for 'soon' is supplied.
Counts as supported if: Contemporaneous research establishes substantial unresolved barriers and provides a sufficiently comprehensive basis for concluding that no demonstrated approach justified expecting a practical solution in the near term.
Counts as contradicted if: Research available before publication demonstrates a broadly effective solution, or a validated approach with concrete grounds for expecting near-term deployment that would resolve the practical search problem.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
A study published April 19, 2023 evaluated search responses collected in late February through late March. Across the tested systems, 51.5% of generated statements were fully supported by citations; this measured citation support, not factual accuracy [e1]. The RARR revision available May 31, 2023 reported improved attribution, but also retrieval errors and failures to update reasoning after factual corrections [e2]. These studies establish citation shortcomings and partial mitigation. They do not settle whether persuasive grounds for expecting a practical search solution soon existed by August 12, 2023 [e1, e2].
Limitations
- The two studies do not comprehensively assess the leading approaches or their prospects for near-term deployment.
- Citation support does not directly measure factual accuracy.
- The claim specifies neither a duration for 'soon' nor an acceptable error rate.
Evidence (2 items)
Context only: arXiv; Nelson F. Liu, Tianyi Zhang, and Percy Liang,
highest precision (89.5)
Why it matters here: Responses were collected from late February through late March. The citation-support shortcomings establish an existing search problem, but do not establish that a near-term solution lacked credible grounds.
- Publisher
- arXiv; Nelson F. Liu, Tianyi Zhang, and Percy Liang
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Luyu Gao and colleagues,
make them attributable
Why it matters here: This research, initially released in 2022, supplies a concrete method and reports improved attribution. Its remaining reasoning and retrieval limitations prevent treating that progress as a demonstrated solution to open-ended search hallucinations.
- Publisher
- arXiv; Luyu Gao and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Research publicly available by August 12, 2023 that comprehensively assesses leading approaches and their deployment prospects, or validates a broadly reliable search approach with concrete grounds for near-term deployment. The assessment would also need an explicit interpretation of 'soon' and 'solved'.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, What if Generative AI turned out to be a Dud?
Perhaps the only use case that really seemed compelling to either of us economically was search (e.g., using Bing powered by ChatGPT instead of Google Search) — but the technical problems there are immense; there is no reason to think that the hallucination problem will be solved soon.
- Where
- paragraph 8
- Published
- 2023-08-13
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-what-if-generative-ai-turned-out-07e6c8-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-what-if-generative-ai-turned-out-07e6c8-a2.