“The real takeaway from the Apple paper is that LLMs can’t be trusted to run algorithms as complexity and distance from the training distribution grows (just as humans shouldn’t serve as calculators).”
Our reading, the sentence that was checked:
LLMs cannot be trusted to run algorithms as problem complexity and distance from the training distribution grow.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Seven replies to the viral Apple reasoning paper – and why they fall short, paragraph 5.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-seven-replies-to-the-viral-apple-c8a2fd-a1
We recorded 17 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-06-12. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- as complexity and distance from the training distribution grows (qualifier, stated by the speaker)
From the speaker
- LLMs can’t be trusted to run algorithms
- as complexity and distance from the training distribution grows
- The real takeaway from the Apple paper
Our interpretation
- The plain reading is a warning about dependable, unaided algorithm execution by contemporary models, not a claim that every model fails every algorithm.
- Both increasing problem complexity and departure from training examples matter to the stated claim.
- Use June 2025, the reporting period of the Apple study discussed in the post. This is an inferred period for an ongoing observation.
Counts as supported if: Controlled evaluations of contemporary models show substantial execution failures as complexity increases and as inputs depart from a documented training distribution, with solvability and output budgets checked.
Counts as contradicted if: Comparable evaluations establish dependable execution across increasing complexity and documented distribution shifts, sufficiently broadly to defeat the general reliability warning.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Apple supplies evidence for the complexity-related reliability concern. It does not directly measure the training-distribution part of Marcus’s broader statement. Earlier controlled studies show both failures and conditional successes in length generalization. The full proposition therefore remains unsettled; this is not a finding that the observed failures were absent.
Limitations
- A narrower claim confined to execution failures in Apple’s tested puzzles would have stronger support.
- These results do not establish permanent limits on future models or systems using external tools.
Evidence (3 items)
Context only: arXiv; Shojaee and colleagues,
limitations.
Why it matters here: The study reports complexity-related failures, including with an algorithm supplied. Its narrow puzzle coverage and black-box access do not establish a measured relationship between accuracy and distance from the proprietary training distributions.
- Publisher
- arXiv; Shojaee and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Anil and colleagues,
significant generalization deficiencies
Why it matters here: Controlled length-generalization results support the concern for earlier models. They do not directly measure the June 2025 models, and the paper also reports improvements from scratchpad prompting.
- Where
- Abstract
- Publisher
- arXiv; Anil and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Huang and colleagues,
possibility of length generalization
Why it matters here: The work supplies theoretical conditions and empirical tests predicting both successful and unsuccessful generalization. It limits any universal reading, but does not demonstrate dependable execution by the frontier models in Apple’s experiment.
- Where
- Abstract
- Publisher
- arXiv; Huang and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Controlled evaluations of the contemporaneous models with documented training exposure, increasing complexity, valid tasks, adequate output budgets, and a stated reliability criterion.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Seven replies to the viral Apple reasoning paper – and why they fall short
The real takeaway from the Apple paper is that LLMs can’t be trusted to run algorithms as complexity and distance from the training distribution grows (just as humans shouldn’t serve as calculators).
- Where
- paragraph 5
- Published
- 2025-06-12
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-seven-replies-to-the-viral-apple-c8a2fd-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.