“Because of the immense amounts of data they are now trained on, and because of the immense number of parameters on which they are trained, that reconstruction can in fact sometimes come very close to memorization.”
Our reading, the sentence that was checked:
Because of their huge training data and parameter counts, LLM reconstruction of inputs sometimes comes very close to memorization.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Deconstructing Geoffrey Hinton’s weakest argument, paragraph 19.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-deconstructing-geoffrey-hintons-weakest-9041dd-a1
We recorded 10 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-02-05. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- due to immense training data and parameter counts (qualifier, stated by the speaker)
From the speaker
- Because of the immense amounts of data
- because of the immense number of parameters
- sometimes
- come very close to memorization
Our interpretation
- Read memorization as near-verbatim or verbatim reproduction of training passages, not reproduction of an arbitrary current prompt.
- The claim includes both the existence of such reproduction and a causal contribution from training-data scale and parameter count.
- Distinguish total corpus size, repeated exposure to a passage and number of training steps.
- Do not require memorization to increase monotonically under every possible training configuration.
Counts as supported if: Extraction research establishes reproduction of training passages, and controlled evidence establishes contributions from parameter count and training-data scale under relevant conditions.
Counts as contradicted if: Evidence rules out the claimed reproduction or materially refutes the alleged causal contributions. Failure to isolate one contribution leaves the compound claim unresolved.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Training-text reproduction is established, as is a strong relationship with parameter count. The compound causal claim goes further by attributing reproduction to the amount of training data. The strongest cited extraction research measures duplication and training exposure, which do not isolate total corpus volume. Later research complicates that explanation but uses a different memorization measure. The whole claim is therefore unresolved; its memorization and model-size components are supported.
Limitations
- This outcome concerns the full causal sentence, not whether LLMs memorize.
- Reading 'amounts of data' as repeated exposure would make the evidence more favorable.
- A missing causal link is not contradictory evidence, so a mixed verdict is not warranted.
Evidence (3 items)
Supports the claim: ICLR 2023; arXiv manuscript,
Larger models memorize
Why it matters here: Supports the parameter-count contribution using exact training-text completion. The study separately measures duplication and prompt length; neither is equivalent to total corpus volume.
- Publisher
- ICLR 2023; arXiv manuscript
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Supports the claim: arXiv,
memorized outputs
Why it matters here: The extraction experiments establish that contemporary models can emit long training passages. Cross-family observations about longer training do not independently isolate the effect of total training-data volume.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv,
memorization from generalization.
Why it matters here: The authors report that unintended memorization can decrease as dataset size grows and models generalize. This complicates a simple data-volume explanation, but its information-based measure and later experiments do not directly contradict the post's passage-reproduction claim.
- Where
- Abstract
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Controlled passage-extraction experiments varying unique corpus volume and parameter count separately while accounting for duplication, training duration and prompting.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Deconstructing Geoffrey Hinton’s weakest argument
Because of the immense amounts of data they are now trained on, and because of the immense number of parameters on which they are trained, that reconstruction can in fact sometimes come very close to memorization.
- Where
- paragraph 19
- Published
- 2024-02-05
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-deconstructing-geoffrey-hintons-weakest-9041dd-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-deconstructing-geoffrey-hintons-weakest-9041dd-a1.