“While that sounds impressive, it’s worth remembering that this is just another benchmark, and thus, is susceptible to the same kind of exploitation as any other benchmark.”
Our reading:
OpenAI's rising IMO scores are just another benchmark susceptible to exploitation.
Assessment: Supported
What happened
Was it true for the period described?
Outcome: Supported
The evidence matches what they said.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
The susceptibility claim is supported for reused, published IMO problems: the November 7, 2024 FrontierMath paper explains that benchmark problems entering training data can inflate scores, while fresh IMO problems created after training avoid that exposure [e1]. This supports a vulnerability, without establishing that contamination caused OpenAI's results [e1]. A study released November 9, 2024 compared o1-mini on public IMO problems and less accessible competition problems and found no significant evidence of reliance on memorization [e2]. OpenAI's September 12, 2024 disclosure identifies its competition-math evaluation as AIME, leaving uncertainty about the IMO evaluation meant here [e3].
Caveats
- The evidence does not establish contamination or score inflation in the particular OpenAI results discussed [e1, e2].
- The supported scope is reuse of published problems; fresh problems created after training avoid the exposure described in e1 [e1].
- The AIME disclosure does not establish which IMO problem set or protocol the claim concerns [e3].
Evidence (3 items)
Supports the claim: arXiv; FrontierMath research team,
inflated performance metrics
Why it matters: Section 5 explicitly identifies contamination risks from historical competition collections, including IMO materials; applying that documented mechanism supports susceptibility when published problems are reused. Section 1 also explains that fresh problems created after training avoid this exposure, limiting the conclusion's scope.
- Publisher
- arXiv; FrontierMath research team
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
Context only: arXiv; Leo Li, Ye Luo and Tingyou Pan,
no significant evidence
Why it matters: The study compared public IMO problems with less accessible Chinese team-training problems and did not find a significant memorization advantage for o1-mini. This weighs against attributing actual performance to memorization, but does not establish that the benchmark cannot be exploited.
- Publisher
- arXiv; Leo Li, Ye Luo and Tingyou Pan
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
Context only: OpenAI,
performance on AIME
Why it matters: OpenAI names AIME in its contemporaneous disclosure. This flags uncertainty about Zitron's benchmark label and is not used as decisive evidence about IMO contamination.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- About events on
- Published
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Where we had to interpret
- The ordinary reading is that evaluation on known IMO problems can reward prior exposure to their answers, rather than independently measuring mathematical ability.
- The claim concerns susceptibility. It does not assert that OpenAI cheated, that contamination explains its improvements, or that every benchmark has equal exposure.
- The target covers the previously published evaluations discussed by the post, through 2025-03-24. The passage supplies no starting date; none is imposed.
- The assessment distinguishes reuse of published problems from testing on fresh problems created after training. The supported reading concerns the former.
- The passage does not identify a particular IMO problem set or evaluation protocol. Evidence about AIME is treated as context, without substituting it for IMO.
Right if: Evidence establishes that published IMO problems used for model evaluation can be encountered during training or targeted preparation, allowing answer-specific familiarity to influence scores. Documentation establishing this vulnerability is sufficient; actual misconduct need not be demonstrated.
Wrong if: Evidence establishes that the referenced evaluation used protected, previously unseen problems under a protocol that excluded prior answer exposure and answer-specific optimization. Failure to detect memorization in one model would not alone establish immunity.
We wrote this test after the outcome was known.
We found 27 claims in this post. See them all.
Full record
Source and dates
Where's Your Ed At, The Phony Comforts of AI Optimism, paragraph 22.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-28. The post may have been edited after 2025-03-24.
Where it was said (1)
Said once in the posts we read.
: Where's Your Ed At, The Phony Comforts of AI Optimism
While that sounds impressive, it’s worth remembering that this is just another benchmark, and thus, is susceptible to the same kind of exploitation as any other benchmark.
- Where
- paragraph 22
- Published
- 2025-03-24
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Supported. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
A reviewer said our test wasn’t fair to what was actually said.