“Much as I admire the new pair of systems, they do have a serious shortcoming and one that is not unfamiliar. The systems both rely on a kind of cheat on the input: trained human coders translate all the Olympiad's input sentences into mathematical form.”
Our reading, the sentence that was checked:
AlphaProof and AlphaGeometry 2 rely on human coders translating Olympiad problems into formal form, a serious shortcoming.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic, paragraph 60.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-alphaproof-alphageometry-chatgpt-22368d-a1
We recorded 17 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-07-28. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- The systems both rely
- trained human coders translate
- all the Olympiad's input sentences
- into mathematical form
Our interpretation
- The claim concerns AlphaProof and AlphaGeometry 2 in the July 2024 IMO demonstration described in the post.
- Human coders means people with the expertise to prepare the formal problem representations; no particular credential is required.
- Input sentences means the competition problem statements supplied to the systems. It does not mean every training example or proof step.
- 'Cheat' and 'serious shortcoming' are evaluations of the workflow, not separate claims that competition rules were violated.
- The observation does not assert that later versions could never automate formalization.
Counts as supported if: Records of the July 2024 demonstration establish that humans converted the competition problem statements into the formal representations used by both systems before automated solving.
Counts as contradicted if: Records establish that either system autonomously translated its actual IMO evaluation inputs from the original natural-language statements without human formalization.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The 2024 demonstration used human-prepared formal problem statements. The primary AlphaProof paper expressly records expert manual formalization, and contemporaneous reporting describes human input preparation for the combined demonstration. This supports Marcus's factual point about the workflow. It does not establish that the assistance was concealed or prohibited, or independently settle his judgment that it was a 'cheat'.
Limitations
- The assessment concerns the July 2024 demonstration, not later systems.
- The evidence for the combined workflow partly relies on a news report because the original announcement was not recoverable through read_page.
- The assessment does not equate human formalization with a rule violation.
Evidence (2 items)
Supports the claim: Nature,
by domain experts.
Why it matters here: The primary paper explicitly confirms expert manual formalization of AlphaProof's competition inputs. Its publication is later, but the reported event is the July 2024 evaluation.
- Publisher
- Nature
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Supports the claim: Ars Technica,
by humans.
Why it matters here: The report covers the combined AlphaProof and AlphaGeometry 2 demonstration and attributes confirmation of human formalization to an evaluator. It supplements the primary AlphaProof protocol because read_page did not recover the original announcement's body.
- Publisher
- Ars Technica
- Kind of source
- News report
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Much as I admire the new pair of systems, they do have a serious shortcoming and one that is not unfamiliar. The systems both rely on a kind of cheat on the input: trained human coders translate all the Olympiad's input sentences into mathematical form.
- Where
- paragraph 60
- Published
- 2024-07-28
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-alphaproof-alphageometry-chatgpt-22368d-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-alphaproof-alphageometry-chatgpt-22368d-a1.