“For one thing, what we publish often gets incorporated into the training; perhaps in company benchmarks as well.”
Our reading, the sentence that was checked:
What the authors publish often gets incorporated into AI training data, and perhaps into company benchmarks as well.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, How Not to Test GPT-3, paragraph 40.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-how-not-to-test-gpt-3-6818ec-a2
We recorded 12 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-02-17. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- what we publish (scope, stated by the speaker)
- perhaps (company benchmarks) (qualifier, stated by the speaker)
From the speaker
- what we publish
- often gets incorporated into the training
- perhaps in company benchmarks as well
Our interpretation
- The firm assertion concerns repeated inclusion of Marcus and Davis's published material, especially their test examples, in LLM training.
- General evidence that training corpora contain published material does not establish this author-specific assertion.
- The benchmark clause expresses a possibility and is not a separate firm assertion.
- I use early 2023 through the post date as the latest reportable period for this ongoing, undated observation.
Counts as supported if: Training records, corpus audits, or validated membership tests establish repeated inclusion of the authors' published examples in models in use during the target period.
Counts as contradicted if: Sufficiently broad training records or corpus audits establish that the authors' published examples were generally excluded from the relevant models.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The 2020 GPT-3 analysis examined overlap between training data and benchmarks, but did not establish repeated inclusion or general exclusion of Marcus and Davis's published examples [e1]. It therefore does not settle the author-specific training claim for January 1–February 17, 2023.
Limitations
- No verified membership records or frequency estimate establish how often the authors' examples entered relevant training data.
- The evidence does not document inclusion in company benchmarks; that clause is a possibility, not a separate firm assertion.
- The absence of membership evidence does not establish exclusion.
Evidence (1 item)
Context only: arXiv; Brown et al.,
overestimated contamination
Why it matters here: The developers investigated overlap between training data and benchmarks, while cautioning about interpreting their detection method. This earlier, general analysis does not establish repeated inclusion of Marcus and Davis's published examples.
- Publisher
- arXiv; Brown et al.
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Training disclosures, independent corpus audits or validated membership tests identifying recurring inclusion of the authors' examples in models used during January 1–February 17, 2023, or sufficiently broad records establishing general exclusion, with publications, model versions and training stages identified.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, How Not to Test GPT-3
For one thing, what we publish often gets incorporated into the training; perhaps in company benchmarks as well.
- Where
- paragraph 40
- Published
- 2023-02-17
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-how-not-to-test-gpt-3-6818ec-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.