“But that is because they rely in part on powerful, completely hand-written, symbolic reasoning systems. LLMs are not similarly immune.”
Our reading, the sentence that was checked:
AlphaProof and AlphaGeometry cannot generate incorrect proofs because they rely in part on powerful, completely hand-written symbolic reasoning systems, and LLMs are not similarly immune.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Reports of LLMs mastering math have been greatly exaggerated, paragraph 5.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-reports-of-llms-mastering-math-have-cc7f8b-a1
We recorded 11 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-04-05. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- rely in part on hand-written symbolic reasoning systems (qualifier, stated by the speaker)
From the speaker
- rely in part
- completely hand-written, symbolic reasoning systems
- LLMs are not similarly immune
- they cannot generate an incorrect proof
Our interpretation
- The comparison concerns the systems discussed in connection with the 2024 IMO and ordinary LLM proof generation as understood by the post date.
- I read the correctness claim as applying to completed proofs accepted by the symbolic machinery, assuming its implementation and mathematical premises are sound. It does not apply to every candidate generated during search.
- Hand-written means programmed checking and deduction rules, rather than learned proof-search behavior.
- LLMs means models producing unchecked proofs, not every possible system combining an LLM with a verifier.
Counts as supported if: Technical descriptions establish that symbolic verification or rule-based deduction controls accepted proofs in both systems, while unchecked language-model output lacks that guarantee.
Counts as contradicted if: Technical descriptions show that accepted proofs bypass those checks, or that the claimed correctness protection comes from learned generation alone. A documented invalid accepted proof would challenge the guarantee.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The technical descriptions support the causal distinction: AlphaProof checks proof terms with Lean, and AlphaGeometry derives conclusions through programmed symbolic rules. Unchecked LLM proofs lack this protection. This supports the ordinary technical reading about accepted proofs. The literal claim that the systems cannot generate any incorrect candidate, or cannot fail under any circumstances, goes beyond the evidence.
Limitations
- The result depends on distinguishing accepted proofs from generated candidates.
- Formal correctness depends on the trusted machinery and the statement supplied to it.
- The AlphaProof paper was published after the post but before the evaluation date and describes the 2024 system.
Evidence (2 items)
Supports the claim: Nature,
Lean’s kernel
Why it matters here: The paper describes AlphaProof's use of Lean and attributes proof correctness to kernel verification. It also explains that informal LLM reasoning lacks guaranteed correctness; this supports the mechanism behind Marcus's comparison.
- Publisher
- Nature
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Supports the claim: arXiv,
Deductive Database Arithmetic Reasoning.
Why it matters here: The authors describe fixed deduction rules and a programmed implementation, rather than trusting language-model assertions. Appendix E separately evaluates model-generated proof steps and documents checks for invalid steps, supporting the distinction between learned proposals and symbolic validation.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Reports of LLMs mastering math have been greatly exaggerated
But that is because they rely in part on powerful, completely hand-written, symbolic reasoning systems. LLMs are not similarly immune.
- Where
- paragraph 5
- Published
- 2025-04-05
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-reports-of-llms-mastering-math-have-cc7f8b-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-reports-of-llms-mastering-math-have-cc7f8b-a1.