Elegant and powerful new result that seriously undermines large language models
We recorded 8 claims from this post. 4 were checked against evidence. 4 can’t be checked. 6 more are held. Read the post at its source.
Claims from this post (8)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
Observation
“The shocking (yet, in historical context, see below, unsurprising) answer is no:”
Our reading: LLMs trained on 'A is B' cannot automatically infer that 'B is A'.
No horizon Evidence as of
Limitation: These experiments do not establish impossibility across all models, training procedures or prompting methods [e1].
Assessment: SupportedSaid
Observation
“On made-up facts, in a first experiment, the model was at zero percent correct, on celebrities, in a second experiment, performance was still dismal.”
Our reading: In the Berglund et al. experiments, the model scored zero percent on reversed made-up facts and performed dismally on reversed celebrity facts.
No horizon Evidence as of
Limitation: The other synthetic reversal direction scored 0.1%, so zero should not be generalized to every test.
Assessment: Can’t be checkedSaid
Observation
“None of what I raised then has really been adequately addressed in the intervening decades.”
Our reading: The generalization problems Marcus raised in 2001 about neural networks have not been adequately addressed since.
No horizon
Assessment: SupportedSaid
Observation
“Forty one years later, neural networks (at least of the popular variety) still struggle with this. They still remain pointillistic masses of blurry memory, never as systematic as reasoning machines ought to be.”
Our reading: Popular neural networks still struggle with systematic relational reasoning and are not as systematic as reasoning machines should be.
No horizon Evidence as of
Limitation: The tested models and tasks do not establish that every neural architecture lacks systematic reasoning [e1].
Assessment: Can’t be checkedSaid
Causal claim
“The will to believe in neural networks is frequently so strong that counterevidence is often dismissed or ignored, for much too long.”
Our reading: The will to believe in neural networks is frequently so strong that counterevidence is often dismissed or ignored for much too long.
No horizon
Qualifier: frequently
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“If, after training on virtually the entire internet, you know Tom is Mary Lee‘s son, but can’t figure out without special prompting that Mary Lee therefore is Tom’s mother, you have no business running all the world’s software.”
Our reading: A system that, after training on virtually the entire internet, knows Tom is Mary Lee's son but can't figure out without special prompting that Mary Lee is therefore Tom's mother has no business running all the world's software.
No horizon
Qualifier: after training on virtually the entire internet
Assessment: UnresolvedSaid
Observation
“This symmetry fail is mighty big, a mighty persistent error that has endured for decades.”
Our reading: The symmetry failure is a big, persistent error in neural networks that has endured for decades.
No horizon Evidence as of
Limitation: Failure to locate a matching historical experiment is not evidence that none exists.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Computers should be expected to make optimal use of information given, regardless of human limitations.”
Our reading: Computers should be expected to make optimal use of the information given, regardless of human limitations.
No horizon
Qualifier: regardless of human limitations
Held from this post (6)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews reached different assessments. Our reading, not assessed: In many real-world problems the space of possible examples can never be fully covered, and heavily data-driven systems like LLMs that lack explicit variables and operations over variables are out of luck when they try to extrapolate beyond the space of their training examples; this was true in 2001 and is still true now.
- Attribution is uncertain
- The criteria are ambiguous. Our reading, not assessed: People will eventually realize that AI needs genuinely new ideas, such as neurosymbolic mechanisms or different approaches.
- The criteria are ambiguous. Our reading, not assessed: It is just a matter [of time] before people start to realize that the field needs some genuinely new ideas, either new mechanisms (perhaps neurosymbolic) or different approaches altogether.
- The record does not meet the current rules. Our reading, not assessed: Bing's neurosymbolic supplementation to pure LLMs gives some benefit on these reversal problems where there are known facts, though it is not yet clear how general the benefit is, and Marcus would expect Bing to have trouble on the made-up facts used in Experiment 1.
- The two reviews read the claim differently