Let’s be honest, Generative AI isn’t going all that well
We recorded 7 claims from this post. 4 were checked against evidence. 3 can’t be checked. Read the post at its source.
Claims from this post (7)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
Observation
“LLMs can still cannot be trusted:”
Our reading: LLMs still cannot be trusted.
No horizon Evidence as of
Limitation: The supplied AA-Omniscience abstract does not specify whether browsing was enabled [e1].
Assessment: UnresolvedSaid
Observation
“A large fraction of what LLMs do is mostly just memorization (and Hinton was on the wrong side of this argument):”
Our reading: A large fraction of what LLMs do is mostly memorization, and Hinton was wrong on this point.
No horizon Evidence as of
Limitation: Large fraction has no stated numerical threshold.
Assessment: UnresolvedSaid
Observation
“They still aren’t adding a lot of quantifiable value to the world:”
Our reading: LLMs still are not adding much quantifiable value to the world.
No horizon Evidence as of
Limitation: The claim lacks a threshold and an agreed measure of value.
Assessment: SupportedSaid
Observation
“Update: This is consistent with the finding of the Remote Labor Index that AI could only do about 2.5% of jobs, reported recently by the Washington Post.”
Our reading: Per the Remote Labor Index, AI can do only about 2.5% of jobs, consistent with LLMs adding little quantifiable value.
No horizon Evidence as of
Limitation: The benchmark result does not establish the share of all occupations AI can perform or aggregate economic value from LLM use [e1].
Assessment: Can’t be checkedSaid
Observation
“Scaling isn’t going all that well, anymore, and probably won’t cure these problems.”
Our reading: Scaling of LLMs is no longer going all that well.
No horizon
Assessment: Can’t be checkedSaid
Forecast
“Scaling isn’t going all that well, anymore, and probably won’t cure these problems.”
Our reading: Scaling will probably not cure LLMs' problems of untrustworthiness, memorization, and limited value.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Trying to orient our economy and geopolitical policy around such shoddy technology — particularly on the unproven hopes that it will dramatically improve– is a mistake.”
Our reading: The economy and geopolitical policy should not be oriented around generative AI, especially on unproven hopes that it will dramatically improve.
No horizon
Qualifier: particularly on unproven hopes of dramatic improvement