“It is truly embarrassing that LLMs cannot reliably solve Hanoi.”
Our reading, the sentence that was checked:
LLMs cannot reliably solve the Tower of Hanoi.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, A knockout blow for LLMs?, paragraph 19.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-a-knockout-blow-for-llms-c79e0c-a3
We recorded 19 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-06-07. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- LLMs cannot reliably solve Hanoi
Our interpretation
- The factual claim concerns reliability; 'truly embarrassing' is an opinion and is excluded.
- In context, reliability means consistently producing complete legal move sequences across the puzzle sizes discussed, including harder instances.
- The claim does not mean that no LLM can solve any Hanoi instance.
- The surrounding discussion distinguishes direct model output from writing and executing conventional code. I assess direct move generation.
- For this ongoing-state claim, the latest relevant reporting period is the June 2025 study snapshot available at the post, not subsequent model releases or experiments.
Counts as supported if: Contemporary evaluations show substantial, repeated failures to produce complete legal Hanoi solutions as puzzle size increases under the discussed direct-generation conditions.
Counts as contradicted if: Comparable evaluations show consistently correct complete solutions across the discussed sizes and conditions.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The June 2025 Apple study reports repeated failures to produce complete legal Hanoi solutions as puzzle size increases, supporting the reliability claim for direct move generation in the evaluated settings [e1]. It also reports near-perfect accuracy for Claude 3.7 Sonnet on five-disc Hanoi, so the finding does not mean that all instances fail [e1]. A follow-up available June 16 reports successful recursive-program generation for 15 discs, but that changes the required output and does not overturn the direct-generation result [e2].
Limitations
- The evidence does not establish failure for every model, prompt, or tool configuration.
- The results do not establish a permanent capability limit or uniquely identify the cause of failure.
- The follow-up's program-generation results are preliminary and lack a highly powered statistical sample [e2].
Evidence (2 items)
Supports the claim: Apple researchers, arXiv,
accuracy is zero.
Why it matters here: The study reports repeated failures in complete solutions and intermediate attempts as complexity increases. This supports unreliable direct Hanoi execution in the evaluated setting, while successful easier cases limit the scope.
- Publisher
- Apple researchers, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: A. Lawsen, arXiv,
The generated solutions correctly implement the recursive algorithm
Why it matters here: The follow-up reports successful program generation for larger Hanoi instances. It changes the required output and follows the post, so it limits an unrestricted inability claim without contradicting the original direct-generation result.
- Publisher
- A. Lawsen, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, A knockout blow for LLMs?
It is truly embarrassing that LLMs cannot reliably solve Hanoi.
- Where
- paragraph 19
- Published
- 2025-06-07
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-a-knockout-blow-for-llms-c79e0c-a3: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.