There Must be Some Misunderstanding
We recorded 7 claims from this post. 4 were checked against evidence. 3 can’t be checked. Read the post at its source.
Claims from this post (7)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Observation
“These systems don’t deeply understand their inputs, contra Dr. Hinton, and as pointed out here regularly.”
Our reading: Large language models do not deeply understand their inputs.
No horizon
Assessment: SupportedSaid
Observation
“Compositionality remains a problem. So does negation.”
Our reading: Compositionality and negation remain problems for large language models.
No horizon Evidence as of
Limitation: Results depend on task, model version and prompt.
Assessment: SupportedSaid
Observation
“Reliability remain a problem.”
Our reading: Reliability remains a problem for large language models.
No horizon Evidence as of
Limitation: Benchmark performance varies by task and scoring method.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“You shouldn’t count on these systems.”
Our reading: People should not rely on large language model systems.
No horizon
Assessment: SupportedSaid
Observation
“Scale isn’t everything, and hasn’t and won’t be enough to solve any of the problems I just listed above.”
Our reading: Scaling has not been enough to solve the problems of understanding, compositionality, negation, and reliability in LLMs.
No horizon Evidence as of
Limitation: The result depends on treating understanding as observable interpretation and generalization.
Assessment: UnresolvedSaid
Forecast
“Scale isn’t everything, and hasn’t and won’t be enough to solve any of the problems I just listed above.”
Our reading: Scaling will not be enough to solve the problems of understanding, compositionality, negation, and reliability in LLMs.
No deadline stated Evidence as of
Limitation: The boundary between scaling and a new method is unclear.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“We really should consider trying some other approaches to AI.”
Our reading: The AI field should consider approaches to AI other than scaling large language models.
No horizon