“In sum, performance on hard coding problems, an alleged area of LLM strength, drops to zero when you control nicely for contamination.”
Our reading, the sentence that was checked:
When data contamination is controlled for, LLM performance on hard coding problems drops to zero.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Five quick updates about that Apple reasoning paper that people can’t stop talking about, paragraph 12.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-five-quick-updates-about-that-apple-977b1e-a1
We recorded 9 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-06-17. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- when controlling for data contamination (qualifier, stated by the speaker)
From the speaker
- performance on hard coding problems
- drops to zero
- when you control nicely for contamination
Our interpretation
- The immediately preceding discussion identifies the original LiveCodeBench Pro paper as the scope.
- Read hard as the benchmark's Hard tier, and performance as its principal pass@1 measure for the evaluated models without external tools.
- Zero means a reported zero success rate, not merely low performance.
- Read when as describing the evaluation conditions. The claim does not establish that removing contamination alone caused the difference.
Counts as supported if: The original paper reports zero pass@1 for every evaluated model on its Hard tier under its contamination-reduction protocol.
Counts as contradicted if: At least one evaluated model has a positive reported pass@1 on that same Hard tier under the same protocol.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The original benchmark reports zero for every listed model on its Hard tier. This supports Marcus's summary within the named benchmark and evaluation settings.
Limitations
- The result concerns the original tested systems without external tools.
- It does not establish zero performance on all difficult coding tasks.
- Contamination reduction is an evaluation safeguard, not a demonstrated sole cause.
Evidence (1 item)
Supports the claim: Zihan Zheng and colleagues, arXiv,
All variants show 0% pass rate on the hard tier in the evaluation.
Why it matters here: Table 1 reports 0.0% Hard pass@1 for every listed model. The quoted caption additionally confirms zero for the tested o4-mini variants with multiple attempts; the protocol reduces contamination and excludes external tools.
- Publisher
- Zihan Zheng and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Five quick updates about that Apple reasoning paper that people can’t stop talking about
In sum, performance on hard coding problems, an alleged area of LLM strength, drops to zero when you control nicely for contamination.
- Where
- paragraph 12
- Published
- 2025-06-17
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-five-quick-updates-about-that-apple-977b1e-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-five-quick-updates-about-that-apple-977b1e-a1.