Seven replies to the viral Apple reasoning paper – and why they fall short
We recorded 13 claims from this post. 4 were checked against evidence. 9 can’t be checked. 2 more are held. Read the post at its source.
Claims from this post (13)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“If we want to get to AGI, we will have to better.”
Our reading: Reaching AGI will require doing better than current LLMs.
No horizon
Qualifier: if we want to get to AGI
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“The length limit on LLM is a bug, and most certainly not a feature.”
Our reading: The output length limit of LLMs is a bug, and most certainly not a feature.
No horizon
Qualifier: most certainly
Assessment: Can’t be checkedSaid
Causal claim
“One is left simply having to test everything, all the time, with little guarantees of anything. Some model might be big enough for task T of size S and fail on the next size, or on Task T’ that is slightly different, etc. It all becomes a crapshoot.”
Our reading: Because some model might be big enough for a task of one size yet fail on the next size or on a slightly different task, one is left having to test everything all the time with little guarantee of anything, making it all a crapshoot.
No horizon
Scope: when relying on bigger models
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Huge vindication for what I have been saying all along: we need AI that integrates both neural networks and symbolic algorithms and representations (such as logic, code, knowledge graphs, etc).”
Our reading: AI should integrate neural networks with symbolic algorithms and representations.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“True in some cases, and a huge win for neurosymbolic AI, given that they can’t reliably solve the puzzles without code, and given that code is symbolic.”
Our reading: That the systems can solve the puzzles with code, which is true in some cases, is a huge win for neurosymbolic AI, given that they can't reliably solve the puzzles without code and code is symbolic.
No horizon
Scope: true in some cases
Assessment: UnresolvedSaid
Causal claim
“But downloading code without conceptual understanding is of less help in the case of new problems, dynamically changing environments, and so on.”
Our reading: LLMs retrieving code without conceptual understanding help less on new problems and dynamically changing environments.
No horizon Evidence as of
Limitation: The meaning of conceptual understanding remains unsettled in this claim.
Assessment: UnresolvedSaid
Observation
“But also, we need to do so reliably, and in a general way and we haven’t yet crossed that threshold.”
Our reading: Current AI does not yet integrate neural and symbolic methods reliably and generally.
No horizon Evidence as of
Limitation: The claim supplies neither a reliability measure nor a required range of tasks.
Assessment: SupportedSaid
Forecast
“I doubt any of them are perfect, but the four together provide converging evidence with dozens of other prior papers (including some of my own), and I am confident many more examples will be uncovered.”
Our reading: Many more examples of LLM/LRM failures at applying algorithms, like those in the Apple paper, will be uncovered.
Settled before its deadline Evidence as of
Limitation: The supporting records are public preprints; publication establishes that examples were reported, not universal acceptance of their explanations.
Assessment: UnresolvedSaid
Forecast
“Mark my words: in time, we will see scores of papers reinforcing the Apple results.”
Our reading: Over time, scores of papers will reinforce the Apple paper's findings on reasoning model limitations.
No deadline stated Evidence as of
Limitation: The bibliography search is not exhaustive.
Assessment: Can’t be checkedSaid
Observation
“The real news here, aside from the fact that this was a clever study nailing down an important point, is that people are finally starting to pay attention, to (one of the) two biggest Achilles’ Heels of generative AI, and to appreciate its significance.”
Our reading: People are finally starting to pay attention to poor generalization, one of the two biggest Achilles' heels of generative AI, and to appreciate its significance.
No horizon
Qualifier: (one of the) two biggest Achilles' heels
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“If people like Sam Altman are sweating, it’s because they should.”
Our reading: If people like Sam Altman are sweating, it is because they should be.
No horizon
Only if: if people like Sam Altman are sweating
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“None of the rejoinders are compelling.”
Our reading: None of the roughly seven rejoinders to the Apple paper reviewed in this post are compelling.
No horizon
Scope: the rejoinders reviewed in this post
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Taking the SalesForce report together with the Apple paper, it’s clear the current tech is not to be trusted.”
Our reading: Taking the Salesforce report together with the Apple paper, it is clear that the current tech is not to be trusted.
No horizon
Qualifier: taking the Salesforce report together with the Apple paper
Held from this post (2)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: In many cases LLMs are a step backwards compared with existing systems that run such algorithms without errors, whereas AGI should be a step forward.