“In recent years, as the financial stakes of benchmark performance have increased, systems have been trained more often directly on benchmarks (“teaching to the test”) or on synthetic data closely resembling them.”
Our reading, the sentence that was checked:
In recent years, as the financial stakes of benchmark performance have increased, AI systems have been trained more often directly on benchmarks or on synthetic data closely resembling them.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Rumors of AGI’s arrival have been greatly exaggerated, paragraph 20.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-rumors-of-agis-arrival-have-been-723913-a1
We recorded 19 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2026-02-17. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- in recent years (scope, stated by the speaker)
- as the financial stakes of benchmark performance have increased (qualifier, stated by the speaker)
From the speaker
- In recent years
- financial stakes of benchmark performance have increased
- trained more often directly on benchmarks
- synthetic data closely resembling them
Our interpretation
- Recent years means the several years preceding publication, operationalized as 2020 through February 17, 2026.
- More often means an increase in the prevalence of this training practice. Individual examples establish occurrence but do not establish an increase.
- Training exposure can be deliberate or accidental; the claim does not require proof that every developer intentionally cheated.
- The financial-stakes phrase describes accompanying circumstances. It does not, by itself, establish financial incentives as the cause.
Counts as supported if: Comparable evidence across time shows that systems increasingly trained on benchmark examples or closely resembling synthetic examples, alongside evidence of increasing commercial stakes.
Counts as contradicted if: Comparable evidence across time shows that such training did not become more common, or that its prevalence declined.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Primary studies establish growing benchmark overlap in web data and actual contamination in a modern training corpus. Those findings make the claim plausible. They do not establish how often successive systems were trained on benchmark material or benchmark-like synthetic data. Treating increasing contamination risk as a measured increase in the training practice would decide the result through an added interpretation.
Limitations
- Public training disclosures are incomplete.
- The evidence establishes occurrence and increasing risk more clearly than increasing prevalence among systems.
Evidence (2 items)
Context only: arXiv; Yucheng Li, Frank Guerin and Chenghua Lin,
increasing rapidly over time.
Why it matters here: The paper compares benchmark overlap in Common Crawl from December 2020 through October 2023. This supports increasing contamination risk, but does not directly measure the proportion of successive model cohorts trained on those examples.
- Publisher
- arXiv; Yucheng Li, Frank Guerin and Chenghua Lin
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Ari Spiesberger and colleagues,
semantic duplicates.
Why it matters here: The researchers find benchmark duplicates in Olmo3 training data and experimentally show that duplicate-based training can improve benchmark scores. A single model family and constructed training experiments do not establish an industry-wide increase over time.
- Publisher
- arXiv; Ari Spiesberger and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
A consistent comparison of dated model training datasets or disclosures showing whether benchmark-based training became more prevalent, plus evidence documenting the accompanying commercial stakes.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Rumors of AGI’s arrival have been greatly exaggerated
In recent years, as the financial stakes of benchmark performance have increased, systems have been trained more often directly on benchmarks (“teaching to the test”) or on synthetic data closely resembling them.
- Where
- paragraph 20
- Published
- 2026-02-17
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-rumors-of-agis-arrival-have-been-723913-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-rumors-of-agis-arrival-have-been-723913-a1.