“So here’s where we are: pure LLMs can’t reliably apply algorithms (vindicating the line of argument I first developed in the 1990s); if you enhance them with symbolic processes—yielding neurosymbolic systems—they often give better results.”
Our reading, the sentence that was checked:
Pure LLMs cannot reliably apply algorithms, while LLMs enhanced with symbolic processes often perform better.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI, paragraph 126.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-how-o3-and-grok-4-accidentally-vindicated-bddd3d-a1
We recorded 47 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-07-13. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- pure LLMs can’t reliably apply algorithms
- if you enhance them with symbolic processes
- they often give better results
- So here’s where we are
Our interpretation
- This describes contemporary capabilities, not a mathematical impossibility or a prediction about all future LLMs.
- Pure LLMs means models producing answers without an external interpreter, calculator, or symbolic solver. Text-only reasoning models remain within this category.
- Reliably means consistently executing valid procedures across changes in inputs and task complexity. The surrounding text expressly allows success on easy, familiar problems.
- Symbolic processes includes executing generated code and using computer algebra systems. Better results means higher correctness on comparable tasks.
- Often means recurring improvements across tested models or task groups, without implying that every model benefits or specifying a numerical majority.
- Research has no fixed reporting quarter. I use the current year through the post date as the contemporary observation period: January 1–July 13, 2025.
- The parenthetical claim of historical vindication is not treated as a separate assertion about intellectual priority.
Counts as supported if: Contemporary controlled tests show substantial failures in unaided algorithm execution across valid inputs or increasing complexity, and same-model comparisons show recurring correctness improvements when external symbolic execution is enabled.
Counts as contradicted if: Broad controlled tests show consistently reliable unaided execution across varied algorithmic tasks and complexities, or comparable symbolic enhancements generally fail to improve correctness. Isolated successes or isolated tool failures do not alone contradict the contextual claim.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The contextual reading is supported. Apple's prescribed-algorithm tests show substantial execution failures, and ASyMOB finds recurring benefits from code execution in same-model comparisons. The qualifications matter: ASyMOB's frontier models did not benefit from tools, corrected River Crossing tests produced strong results, and models could generate correct compact Hanoi programs. Those findings limit the breadth of the claim without defeating its ordinary reading: contemporary unaided models had substantial reliability limits, and symbolic execution could repeatedly improve results. The evidence does not establish that pure LLMs can never execute algorithms reliably or that symbolic tools are always necessary.
Limitations
- The outcome depends on reading the statement as a contemporary empirical generalization.
- The evidence consists of research preprints and selected task families.
- Symbolic mathematics benchmarks provide broader support for tool benefits than for execution of a prescribed algorithm.
- The assessment does not establish Marcus's intellectual priority or a unique explanation of model internals.
Evidence (4 items)
Supports the claim: Apple researchers, arXiv,
fail to use explicit algorithms
Why it matters here: Providing the Tower of Hanoi algorithm did not remove the tested models' execution failures. This supports a practical reliability limitation, although other parts of the puzzle study face methodological objections.
- Publisher
- Apple researchers, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: Technion researchers, arXiv,
an average improvement of 3.1%.
Why it matters here: ASyMOB compares symbolic mathematics performance with and without code execution and reports an average benefit, especially for weaker models. It also reports substantial perturbation sensitivity, but finds no tool benefit for its frontier models, limiting the generalization.
- Publisher
- Technion researchers, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: Dellibarda Varela and colleagues, arXiv,
solve large instances
Why it matters here: The replication finds strong River Crossing performance after correcting solvability, alongside persistent Hanoi failures under stepwise prompting. This counters a blanket inability interpretation and weakens reliance on Apple's River Crossing results, while leaving cross-task reliability limitations intact.
- Publisher
- Dellibarda Varela and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: A. Lawsen, arXiv,
correctly implement the recursive algorithm
Why it matters here: Preliminary tests found that several models could generate compact Hanoi programs successfully. This challenges claims that execution failures prove an inability to understand algorithms, but generating a program does not establish reliable unaided execution; the author also acknowledges insufficient trials for strong statistical conclusions.
- Publisher
- A. Lawsen, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
So here’s where we are: pure LLMs can’t reliably apply algorithms (vindicating the line of argument I first developed in the 1990s); if you enhance them with symbolic processes—yielding neurosymbolic systems—they often give better results.
- Where
- paragraph 126
- Published
- 2025-07-13
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-how-o3-and-grok-4-accidentally-vindicated-bddd3d-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-how-o3-and-grok-4-accidentally-vindicated-bddd3d-a1.