“I predict that such counterexamples will still be easy to find at the end of 2027 because of limitations inherent in LLMs.”
Our reading, the sentence that was checked:
At the end of 2027 it will still be easy to find clear counterexamples where LLM-based generative AI fails at compositionality, because of limitations inherent in LLMs.
Assessment: Unresolved Not yet due on 2026-10-29 Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Scott Alexander’s Misleading Victory Lap, paragraph 39.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-scott-alexanders-misleading-victory-e7b983-a1
We recorded 6 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-07-09. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Forecast
- Deadline, as the speaker wrote it
- “at the end of 2027”
- Deadline state when assessed
- Not yet due
- Deadline state on 2026-10-29
- Not yet due (the release’s reference date)
Conditions
- because of limitations inherent in LLMs (qualifier, stated by the speaker)
From the speaker
- such counterexamples
- will still be easy to find
- at the end of 2027
- because of limitations inherent in LLMs
Our interpretation
- Such counterexamples refers to clear compositional failures like those discussed in the preceding sentences.
- The ordinary reading concerns contemporary leading LLM-based generators at the end of 2027, rather than continued access to an unchanged 2025 model.
- Easy to find means little search effort by an ordinary informed tester. The preceding six-in-under-an-hour account is a useful reference, not an explicitly repeated numerical requirement.
- Because of limitations inherent in LLMs is a causal explanation asserted by the forecast, not an antecedent that must first occur.
- The latest day meant by the deadline is December 31, 2027.
Counts as supported if: Testing at the end of 2027 readily finds clear compositional failures in contemporary leading LLM-based generators, and comparative technical evidence supports the attribution to inherent LLM limitations.
Counts as contradicted if: Broad, documented end-of-2027 testing shows that such failures are no longer readily elicited, or strong technical evidence refutes the asserted inherent cause of the relevant failures.
Prospective: these criteria were frozen before the deadline, while the outcome was not yet public.
Current assessment
Did the forecast come true?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- Not yet due
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The deadline is December 31, 2027, after the October 28, 2026 evaluation freeze. The reviewed research neither fixes future performance nor establishes that the relevant failures are inherent to LLMs. No event meets the early-resolution rule.
Limitations
- Existing failures cannot establish that comparable failures will remain easy to find at the deadline.
- Successful repairs in one system cannot disprove the broader forecast.
Evidence (1 item)
Context only: Ashritha Gonuguntla, arXiv,
The planner is the bottleneck.
Why it matters here: The study diagnoses planning failures and reports improvements from editing plans in a particular reasoning-augmented generator. It illustrates why present failures do not alone establish an inherent limitation or settle performance at the forecast deadline.
- Publisher
- Ashritha Gonuguntla, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Independent testing of contemporary leading systems near the end of 2027, with documented search effort and clear scoring, plus technical evidence addressing whether the observed limitations are inherent.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Scott Alexander’s Misleading Victory Lap
I predict that such counterexamples will still be easy to find at the end of 2027 because of limitations inherent in LLMs.
- Where
- paragraph 39
- Published
- 2025-07-09
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-scott-alexanders-misleading-victory-e7b983-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-scott-alexanders-misleading-victory-e7b983-a1.