Five quick updates about that Apple reasoning paper that people can’t stop talking about
We recorded 9 claims from this post. 4 were checked against evidence. 5 can’t be checked. 1 more is held. Read the post at its source.
Claims from this post (9)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
Observation
“The core of the paper challenged the idea that LLMs can reason on any deep level, and showed that they broke down on moderately complex versions of basic algorithms (since Tower of Hanoi with 8 discs, first solved with classical techniques around 1957) — casting serious doubt into whether LLMs on their own could ever achieve AGI.”
Our reading: The Apple paper challenged the idea that LLMs can reason on any deep level and showed that they broke down on moderately complex versions of basic algorithms such as Tower of Hanoi with 8 discs, which casts serious doubt on whether LLMs on their own could ever achieve AGI.
No horizon Evidence as of
Limitation: Support applies to the reported experiments and their challenge to reasoning reliability.
Assessment: Can’t be checkedSaid
Observation
“To some extent they do this better than in 1998, but is still the core challenge.”
Our reading: Although neural networks generalize beyond their training data somewhat better than in 1998, distribution shift is still the core challenge for them.
No horizon
Qualifier: generalization beyond training data (distribution shift)
Assessment: SupportedSaid
Observation
““The illusion of the illusion” turned out to be an error-ridden joke.”
Our reading: The rejoinder “The Illusion of the Illusion of Thinking” turned out to be an error-ridden joke.
No horizon Evidence as of
Limitation: This assessment concerns the original rejoinder circulating before June 17, 2025, not the corrected version.
Assessment: SupportedSaid
Observation
“A new paper with a hard coding benchmark called LiveCodeBenchPro designed to resist data contamination shows still more converging evidence for the two core notions behind the Apple paper (and behind my own decades-long critique): (a) the systems are challenged at reasoning, and (b) performance declines as one strays further and further from familiar problems.”
Our reading: LiveCodeBenchPro gives further evidence that LLMs are challenged at reasoning and that their performance declines the further problems are from familiar ones.
No horizon Evidence as of
Limitation: Problem categories are an indirect measure of familiarity.
Assessment: SupportedSaid
Observation
“In sum, performance on hard coding problems, an alleged area of LLM strength, drops to zero when you control nicely for contamination.”
Our reading: When data contamination is controlled for, LLM performance on hard coding problems drops to zero.
No horizon Evidence as of
Limitation: The result concerns the original tested systems without external tools.
Assessment: Can’t be checkedSaid
Observation
“What the Apple paper said still looks to be basically correct.”
Our reading: The Apple reasoning paper's conclusions about LLM reasoning limits still look basically correct.
No horizon
Assessment: Can’t be checkedSaid
Observation
“On familiar problems with no wrinkles, they are great. On anything else they are suspect.”
Our reading: Generative AI systems do well on familiar problems with no wrinkles, but their outputs on anything else are suspect.
No horizon
Qualifier: problems not familiar or with wrinkles
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“If we want to get to AGI, we’ll need to do a lot better.”
Our reading: Reaching AGI will require doing much better than current generative AI on distribution shift.
No horizon
Qualifier: if the goal is AGI
Assessment: Can’t be checkedSaid
Rhetoric
“Gary Marcus is sorry to keep writing about how LLMs struggle with distribution shift, but it’s the whole ballgame.”
Our reading: LLMs struggle with distribution shift, and this is the whole ballgame.
No horizon
Held from this post (1)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The record does not meet the current rules. Our reading, not assessed: After almost three decades of work, generative AI still struggles with distribution shift.