“(a) the tendency of large language models to lose coherence over time, drifting into “empty” language with no clear connection to reality;”
Our reading, the sentence that was checked:
Neglecting reference, cognitive models and compositionality leads LLMs to lose coherence over time and drift into empty language with no clear connection to reality.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Three ideas from linguistics that everyone in AI should know, paragraph 9.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-three-ideas-from-linguistics-that-76a3c8-a1
We recorded 26 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2022-06-22. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- caused by inadequate attention to reference, cognitive models, compositionality (qualifier, stated by the speaker)
From the speaker
- inadequate attention to these three factors
- the tendency of large language models to lose coherence over time
- drifting into “empty” language
- no clear connection to reality
Our interpretation
- The causal introduction applies to this bullet; the assessment therefore retains the proposed explanation instead of grading only the existence of incoherent output.
- Over time means as a generation or discourse continues, rather than deterioration across successive model releases.
- The ordinary measurable outcomes are loss of consistency, relevance and factual grounding over longer generations. Empty language has no separate technical definition.
- The causal phrase is part of the proposition, not an antecedent of a conditional forecast.
Counts as supported if: Controlled changes to reference, persistent state representations or compositional mechanisms establish that deficits in the specified capabilities materially cause worsening coherence and grounding over extended generation.
Counts as contradicted if: Well-controlled tests show that the specified deficits do not materially contribute to length-related coherence and grounding failures, or reliably produce the opposite effect.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Long-generation coherence failures are documented, including by the GPT-3 authors. Planning studies also offer relevant evidence that structured guidance can help. But these findings do not establish that inadequate reference, cognitive models and compositionality caused the failures. The interventions bundle several changes, and story consistency is not the same as factual grounding. The symptom is documented; the stated causal explanation remains unresolved.
Limitations
- TattleTale has limited empirical evaluation.
- Re3 is a later experiment and is treated only as context.
- The post supplies no quantitative definition of empty language.
Evidence (3 items)
Context only: OpenAI; arXiv,
a number of limitations.
Why it matters here: The following paragraph reports semantic repetition, loss of coherence over long passages, contradictions and non-sequiturs. This supports the existence of the symptom, but does not identify Marcus’s proposed causes.
- Publisher
- OpenAI; arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: ICAPS 2022 SPARK Workshop; Nisha Simon and Christian Muise,
generate a narrative.
Why it matters here: TattleTale supplies planner-generated actions to GPT-J-6B and reports better narrative consistency. Its limited evaluation and lack of external factual-grounding tests prevent this from settling the broader causal claim.
- Publisher
- ICAPS 2022 SPARK Workshop; Nisha Simon and Christian Muise
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Context only: arXiv; Kevin Yang, Yuandong Tian, Nanyun Peng and Dan Klein,
greatly constrains our experiments.
Why it matters here: Re3 reports better story coherence from a package of planning, repeated context, reranking and editing around GPT-3-based generation. The experiment postdates the claim and does not isolate the three linguistic factors or external factual grounding.
- Publisher
- arXiv; Kevin Yang, Yuandong Tian, Nanyun Peng and Dan Klein
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Matched interventions on contemporary models that vary the specified capabilities and measure how coherence and factual grounding change with generation length, while controlling for additional context and computation.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Three ideas from linguistics that everyone in AI should know
(a) the tendency of large language models to lose coherence over time, drifting into “empty” language with no clear connection to reality;
- Where
- paragraph 9
- Published
- 2022-06-22
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-three-ideas-from-linguistics-that-76a3c8-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-three-ideas-from-linguistics-that-76a3c8-a1.