“Each new bigger, more expensive model ekes out measurable improvements, but returns appear to be diminishing (that’s what Sutskever is saying about the Kaplan laws) and none of these experiments has solved core issues around hallucinations, generalization, planning and reasoning, as Sutskever too now recognizes.”
Our reading, the sentence that was checked:
No LLM scaling experiment has solved the core problems of hallucination, generalization, planning and reasoning.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, A trillion dollars is a terrible thing to waste, paragraph 19.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-a-trillion-dollars-is-a-terrible-9a927b-2-a2
We recorded 14 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-11-27. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Each new bigger, more expensive model ekes out measurable improvements
- returns appear to be diminishing
- none of these experiments has solved core issues
- hallucinations, generalization, planning and reasoning
- as Sutskever too now recognizes
Our interpretation
- The scored core is the firm part: that no scaled LLM had, by the post date (2025-11-27), solved hallucination, generalization, planning and reasoning. The 'returns appear to be diminishing' part is hedged ('appear') and is treated as context, not scored separately.
- Plain reading of 'solved': the problem has been overcome to the point that it is no longer a basic limitation of frontier models. Steady improvement or strong results on specific benchmarks does not count as 'solved'.
- Target period runs from the start of the Kaplan-laws scaling era (2020) to the post date, because the claim covers all such experiments up to then.
Counts as supported if: Credible evidence from around late 2025 (lab research, independent evaluations, or leading researchers) showing that state-of-the-art LLMs still had substantial hallucination, generalization, planning or reasoning failures.
Counts as contradicted if: Credible evidence that by late 2025 some scaled LLM had effectively eliminated one or more of these problems, for example hallucination rates near zero on open-ended factual tasks, or robust human-level generalization and planning on new tasks.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Read plainly, the claim is that by late November 2025 no scaled LLM had overcome hallucination, generalization, planning and reasoning as basic limits. In September 2025, OpenAI's own researchers wrote that hallucinations 'persist even in state-of-the-art systems' (e1). In June 2025, Apple researchers found that frontier reasoning models suffer 'complete accuracy collapse' on planning puzzles past a certain complexity (e2). Two days before the post, Sutskever said models 'generalize dramatically worse than people' (e3), which confirms the post's attribution. There was real progress, such as gold-medal-level IMO results in July 2025 (e5) and what ARC Prize later called a profound upgrade in reasoning (e4). But neither source claims these problems were solved, and ARC Prize itself says reasoning systems still have many flaws. The evidence fits the claim that improvements were measurable but the core problems remained unsolved.
Limitations
- 'Solved' has no stated threshold. Someone who counts IMO gold or big benchmark gains as solving reasoning would read the claim differently.
- The hedged diminishing-returns part was not scored.
- Some evidence covers particular task families (puzzles, factual recall) rather than the four problem areas as a whole.
Evidence (5 items)
Supports the claim: arXiv (Kalai, Nachum, Vempala, Zhang; OpenAI and Georgia Tech),
Such “hallucinations” persist even in state-of-the-art systems and undermine trust.
Why it matters here: OpenAI's own researchers say hallucination had not been solved in state-of-the-art models as of September 2025.
- Where
- Abstract
- Publisher
- arXiv (Kalai, Nachum, Vempala, Zhang; OpenAI and Georgia Tech)
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: Apple Machine Learning Research,
we show that frontier LRMs face a complete accuracy collapse beyond certain complexities. Moreover, they exhibit a counter-intuitive scaling limit
Why it matters here: Independent tests on planning puzzles such as Tower of Hanoi and Blocks World found that frontier reasoning models fail completely past a certain complexity. That shows reasoning and planning were not solved.
- Where
- Abstract
- Publisher
- Apple Machine Learning Research
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: Dwarkesh Podcast,
these models somehow just generalize dramatically worse than people. It’s super obvious. That seems like a very fundamental thing.
Why it matters here: A leading researcher and OpenAI co-founder says generalization remains a basic, unsolved weakness of current models. This also confirms the post's attribution to Sutskever.
- Publisher
- Dwarkesh Podcast
- Kind of source
- News report
- Describes events of
- Public from
Context only: ARC Prize Foundation,
As good as AI reasoning systems are, they still exhibit many flaws and inefficiencies necessary for AGI. We still need new ideas
Why it matters here: The benchmark operator's view of late-2025 frontier systems agrees that reasoning flaws remain. The same post also calls 2025 reasoning systems 'a profound upgrade'. It was published after the post, so it is context only.
- Publisher
- ARC Prize Foundation
- Kind of source
- Benchmark result
- Describes events of
- Public from
Context only: Simon Willison's Weblog,
Gemini achieved the exact same score as OpenAI, 35/42, and were able to solve the same set of questions - 1 through 5, failing only to answer 6
Why it matters here: Counterevidence on the size of progress: LLM-based systems reached gold-medal level at the 2025 IMO. This is major reasoning progress on one hard task, but it does not show reasoning was solved in general.
- Where
- Paragraph 4
- Publisher
- Simon Willison's Weblog
- Kind of source
- Secondary summary
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, A trillion dollars is a terrible thing to waste
Each new bigger, more expensive model ekes out measurable improvements, but returns appear to be diminishing (that’s what Sutskever is saying about the Kaplan laws) and none of these experiments has solved core issues around hallucinations, generalization, planning and reasoning, as Sutskever too now recognizes.
- Where
- paragraph 19
- Published
- 2025-11-27
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-a-trillion-dollars-is-a-terrible-9a927b-2-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-a-trillion-dollars-is-a-terrible-9a927b-2-a2.