“Humans can write summaries of things without hallucination; LLMs cannot do so reliably.”
Our reading, the sentence that was checked:
LLMs cannot reliably write summaries without hallucinating.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, No, Virginia, AGI is not imminent, paragraph 19.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-no-virginia-agi-is-not-imminent-b7e604-a1
We recorded 17 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-08-20. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Humans can write summaries of things without hallucination
- LLMs cannot do so reliably
Our interpretation
- The plain reading concerns dependable factual faithfulness in ordinary summarization, not an inability to produce any correct summary.
- Hallucination means adding factual content unsupported by, or inconsistent with, the material being summarized.
- For this ongoing observation, I use August 2023 through publication as the latest reporting window. The monthly boundary is an evaluator convention because model evaluations have no fixed reporting cycle.
- No numerical reliability threshold, document distribution, prompting method, or model list is supplied. I do not substitute a requirement of perfect accuracy.
Counts as supported if: Representative evaluations of then-current LLMs show recurring unsupported factual content across ordinary summarization tasks, sufficient to undermine dependable use.
Counts as contradicted if: Representative evaluations show that then-current LLMs dependably produce factually faithful summaries across ordinary tasks, including suitable comparisons with human summaries.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The evidence establishes neither broad reliability nor broad unreliability for the target period. Earlier human evaluations found strong news-summary faithfulness, while a later Vectara snapshot found residual hallucinations. Neither should be relabeled an August measurement. A nonzero error rate alone also does not settle what Marcus meant by reliably.
Limitations
- The inferred monthly observation window is narrower than a general review of 2023 models.
- News benchmarks do not represent every summarization use.
- Automated hallucination scores and human factuality judgments are not interchangeable.
Evidence (2 items)
Context only: arXiv; Tianyi Zhang and colleagues,
Instruction tuned models have strong summarization ability.
Why it matters here: The study reports an XSUM faithfulness score of 0.97 for zero-shot Instruct Davinci, falling to 0.77 with five reference examples. This is material counterevidence to sweeping incapacity, but concerns earlier evaluations and a limited news-summary setting.
- Publisher
- arXiv; Tianyi Zhang and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: Vectara,
hallucinations when summarizing a document.
Why it matters here: The snapshot reports a 3.0% hallucination rate for GPT-4 and 3.5% for GPT-3.5 using Vectara's automated evaluator. These later measurements illustrate both substantial factual consistency and residual errors; they are not August outcome evidence.
- Publisher
- Vectara
- Kind of source
- Benchmark result
- Describes events of
- Public from
What would settle it
Dated evaluations of August 2023 model versions across ordinary summary tasks, with human-checked unsupported-content rates and comparable human baselines. Borderline results would still require an explicit reliability threshold.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, No, Virginia, AGI is not imminent
Humans can write summaries of things without hallucination; LLMs cannot do so reliably.
- Where
- paragraph 19
- Published
- 2023-08-20
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-no-virginia-agi-is-not-imminent-b7e604-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-no-virginia-agi-is-not-imminent-b7e604-a1.