“I doubt any of them are perfect, but the four together provide converging evidence with dozens of other prior papers (including some of my own), and I am confident many more examples will be uncovered.”
Our reading, the sentence that was checked:
Many more examples of LLM/LRM failures at applying algorithms, like those in the Apple paper, will be uncovered.
Assessment: Supported Settled before its deadline on 2026-10-29 Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Seven replies to the viral Apple reasoning paper – and why they fall short, paragraph 10.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-seven-replies-to-the-viral-apple-367767-a1
We recorded 13 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-06-12. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Forecast
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- Settled before its deadline
- Deadline state on 2026-10-29
- Settled before its deadline (the release’s reference date)
From the speaker
- many more examples will be uncovered
- analogous failures in algorithm application
Our interpretation
- The forecast concerns subsequently reported failures to apply rules or algorithms as task complexity, sequence length, or task variation increases.
- Many more means a substantial collection across multiple independent studies and distinct task families, rather than numerous sampled failures of one puzzle.
- This claim counts examples, not papers. It does not require every later study to endorse Apple's interpretation of why failures occur.
- The accompanying statement about dozens of prior papers is not the forecast being assessed here.
Counts as supported if: Post-publication empirical research documents a substantial collection of analogous failures across multiple independent studies and task families.
Counts as contradicted if: There is no finite-date contradiction from a shortage of examples because no deadline was stated. Evidence invalidating alleged examples would remove them from the supporting collection.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Did the forecast come true?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- Settled before its deadline
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Three later independent studies document failures across classical reasoning puzzles, sequential state and relation tracking, and spatial rule execution. Together they satisfy the ordinary reading of many additional examples. This supports the existence forecast, not a claim that all models fail everywhere or that Apple's explanation of every failure is correct. The assessment depends on the stated reading of many.
Limitations
- The supporting records are public preprints; publication establishes that examples were reported, not universal acceptance of their explanations.
- Task families overlap in broad computational demands and should not be added into a single supposedly independent example count.
- Some models and configurations perform substantially better than others.
Evidence (3 items)
Supports the claim: arXiv; Utsho and colleagues,
do not generalize
Why it matters here: The published study reports complexity-related declines across nine validated reasoning tasks, including satisfiability, cryptarithmetic, graph coloring, water jugs, Sudoku, and Rubik's Cube. It supplies additional algorithmic and constraint-following examples beyond Apple's original task set.
- Where
- Abstract
- Publisher
- arXiv; Utsho and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: arXiv; Chapra and colleagues,
attempts fail.
Why it matters here: The study evaluates spatial state tracking, symbolic pointer manipulation, and transitive inference while varying sequential depth. Its failures provide additional examples, although stronger models retain substantial performance in some domains.
- Publisher
- arXiv; Chapra and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: arXiv; Xu and colleagues,
declines sharply
Why it matters here: Grid-Bench tests explicit rules across eight spatial-operation families with executable reference answers. Together with e1 and e2, it establishes a substantial collection of later failures across distinct tasks and independent research teams.
- Publisher
- arXiv; Xu and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Seven replies to the viral Apple reasoning paper – and why they fall short
I doubt any of them are perfect, but the four together provide converging evidence with dozens of other prior papers (including some of my own), and I am confident many more examples will be uncovered.
- Where
- paragraph 10
- Published
- 2025-06-12
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-seven-replies-to-the-viral-apple-367767-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.