Deconstructing Geoffrey Hinton’s weakest argument
We recorded 10 claims from this post. 8 were checked against evidence. 2 can’t be checked. 6 more are held. Read the post at its source.
Claims from this post (10)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: MixedSaid
Observation
“The New York Times versus OpenAI lawsuit made that abundantly clear, with a hundred examples like this one, in which ChatGPT reconstructed entire stretches of text (red) from brief prompts2 (black text in top line):”
Our reading: The NYT v. OpenAI lawsuit showed about a hundred examples of ChatGPT reconstructing entire stretches of text from brief prompts.
No horizon Evidence as of
Limitation: Both filings are party submissions, not judicial findings about experimental validity.
Assessment: UnresolvedSaid
Causal claim
“Because of the immense amounts of data they are now trained on, and because of the immense number of parameters on which they are trained, that reconstruction can in fact sometimes come very close to memorization.”
Our reading: Because of their huge training data and parameter counts, LLM reconstruction of inputs sometimes comes very close to memorization.
No horizon Evidence as of
Limitation: This outcome concerns the full causal sentence, not whether LLMs memorize.
Assessment: SupportedSaid
Observation
“Neural networks do effectively memorize some of what they are trained on, and indeed there is a whole subfield now about the topic.”
Our reading: Neural networks effectively memorize some of what they are trained on, and there is now a whole subfield devoted to this topic.
No horizon Evidence as of
Limitation: For closed models, the researchers used an auxiliary corpus because the complete training data were unavailable.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“By not even acknowledging the growing literature on memorization (let alone challenging it), Hinton made himself look to be seriously out of touch.”
Our reading: By not even acknowledging, let alone challenging, the growing literature on memorization, Hinton made himself look seriously out of touch.
No horizon
Qualifier: by not even acknowledging (let alone challenging) the growing literature on memorization
Assessment: Can’t be checkedSaid
Observation
“One might conceivably attribute some shallow level of understanding to LLMs, but there are many reasons to doubt that current neural networks have deep understanding.”
Our reading: There are many reasons to doubt that current neural networks have deep understanding, although one might conceivably attribute some shallow level of understanding to LLMs.
No horizon
Scope: current neural networks
Assessment: UnresolvedSaid
Causal claim
““Given vast datasets, LLMs approximate well, but their understanding is at best superficial. That’s *why* they are unreliable, and unstable, hallucinate, are constitutionally unable to fact check, etc.””
Our reading: Given vast datasets, LLMs approximate well, but their understanding is at best superficial, and that is why they are unreliable, unstable, hallucinate, are constitutionally unable to fact-check, and so on.
No horizon Evidence as of
Limitation: Partial error detection is not reliable verification of arbitrary claims.
Assessment: UnresolvedSaid
Observation
“LLMs do (sometimes) regurgitate large chunks of effectively stored text, and they don’t understand anything at a deep level.”
Our reading: LLMs sometimes regurgitate large chunks of effectively stored text and do not understand anything at a deep level.
No horizon Evidence as of
Limitation: Closed-model training membership is inferred from matching pre-existing material rather than inspection of the complete private training corpus.
Assessment: SupportedSaid
Observation
“LLMs don’t store things in simple databases, but when presented with enough data, their regeneration of distributed bits has demonstrably duplicated lengthy training materials multiple times, a phenomenon that is called memorization in the field.”
Our reading: Given enough data, LLMs have demonstrably duplicated lengthy training materials multiple times (memorization).
No horizon Evidence as of
Limitation: Extraction methods can involve deliberate prompting and specialized decoding.
Assessment: SupportedSaid
Observation
“But the variety of the examples along with the technical literature on memorization and data leakage in large language models make clear that this kind of reconstruction certainly can happen with some regularity.”
Our reading: Reconstruction of training text by large language models can happen with some regularity.
No horizon Evidence as of
Limitation: Marcus supplies no numerical meaning for 'some regularity'.
Assessment: UnresolvedSaid
Observation
“OpenAI will no doubt by now have filtered this specific example so that it can no longer be replicated, now that it has appeared in a lawsuit.”
Our reading: OpenAI has no doubt by now filtered this specific NYT-lawsuit example so that it can no longer be replicated, now that it has appeared in a lawsuit.
No horizon Evidence as of
Limitation: The relevant generation setup is unspecified.
Held from this post (6)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: LLMs' many failures, such as regular but unpredictable hallucinations, show they are simply approximators without true common sense.
- The criteria are ambiguous. Our reading, not assessed: Except in hallucinations associated with disorders like schizophrenia, humans rarely make things up wholesale, whereas LLMs do.
- The criteria are ambiguous. Our reading, not assessed: Yann LeCun has recently shifted to Marcus's position on LLM understanding.
- The record does not meet the current rules. Our reading, not assessed: The mechanisms behind LLM hallucinations are not the same as those behind a human error like forgetting where one put one's car keys, and the errors themselves are qualitatively different.
- The two reviews read the claim differently