Reports of the birth of AGI are greatly exaggerated
We recorded 11 claims from this post. 4 were checked against evidence. 7 can’t be checked. 8 more are held. Read the post at its source.
Claims from this post (11)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“The problem with the paper though isn’t just its hostility towards critics, or the way in which it lowers the bar, it’s that its key arguments don’t go through.”
Our reading: The problem with the Agüera y Arcas and Norvig paper is not just its hostility toward critics or the way it lowers the bar for AGI, but that its key arguments don't go through.
No horizon
Assessment: UnresolvedSaid
Observation
“The first author’s nine-year-old can do five digit multiplication better than an LLM.”
Our reading: The first author's own nine-year-old child can do five-digit multiplication better than an LLM.
No horizon Evidence as of
Limitation: A private comparison may have occurred.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“It’s certainly plausible that some of the prerequisites of AGI may have been achieved, but considerable overreach to presume that all of “the most important ones” have already been invented.”
Our reading: It is certainly plausible that some prerequisites of AGI may have been achieved, but it is considerable overreach to presume that all of the most important ones have already been invented.
No horizon
Qualifier: certainly plausible that some ... may have been achieved
Assessment: Can’t be checkedSaid
Forecast
“Maybe what’s required already exists, but more likely, we are still at least a few paradigm shifts away .”
Our reading: AGI is more likely than not still at least a few paradigm shifts away.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“The fact that LLMs trained on the task of next-token prediction can also do a pretty good job at a variety of natural language tasks, and, very recently, a considerably less good job of tasks that combine text with images, as emphasized by Agüera y Arcas and Norvig is certainly interesting, but it doesn’t justify the claim that they constitute “general intelligence”.”
Our reading: LLMs trained on next-token prediction doing a pretty good job at a variety of natural language tasks, and very recently a considerably less good job at tasks combining text with images, is certainly interesting but does not justify the claim that they constitute general intelligence.
No horizon
Qualifier: very recently
Assessment: SupportedSaid
Observation
“Transformers and generative AI are getting a lot of attention, and have considerable utility, but they are nothing like the all purpose tools some people seems to imagine them to be.”
Our reading: Transformers and generative AI are getting a lot of attention and have considerable utility, but they are nothing like the all-purpose tools some people seem to imagine them to be.
No horizon Evidence as of
Limitation: The result depends on interpreting all-purpose as broadly reliable practical competence.
Assessment: Can’t be checkedSaid
Forecast
“In AI, no single technique is likely to be a silver bullet, now or ever.”
Our reading: No single AI technique is likely ever to be a silver bullet.
No horizon
Assessment: UnresolvedSaid
Observation
“There are some applications where the current systems are reliable enough to be practically useful, though not nearly as many as is often claimed.”
Our reading: Current systems are reliable enough to be useful in some applications, but far fewer than often claimed.
No horizon Evidence as of
Limitation: The existence of useful assistance is easier to test than the claimed extent of overstatement.
Assessment: SupportedSaid
Forecast
“No doubt the next generation of systems will have greater abilities and more extensive applications and will somewhat mitigate the weaknesses.”
Our reading: The next generation of systems will, without doubt, have greater abilities and more extensive applications and will somewhat mitigate the weaknesses of current systems.
Settled before its deadline Evidence as of
Limitation: The evidence is concentrated in the Claude successor family.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“But there is zero justification for claiming that the current technology has achieved general intelligence.”
Our reading: There is zero justification for claiming that the current technology has achieved general intelligence.
No horizon
Assessment: Can’t be checkedSaid
Observation
“Current models trained on next-token prediction have remarkable abilities and remarkable weaknesses. The scope of those abilities and weaknesses of the current systems is not well understood.”
Our reading: Current next-token prediction models have remarkable abilities and weaknesses whose scope is not well understood.
No horizon
Held from this post (8)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The record does not meet the current rules. Our reading, not assessed: Current AI systems lack intelligence that is flexible and general with resourcefulness and reliability comparable to humans, especially reliability.
- The record does not meet the current rules. Our reading, not assessed: LLMs are weak at planning and at maintaining world models, which strongly suggests key ingredients of AGI have not yet been invented.
- The two reviews reached different assessments. Our reading, not assessed: By the classical criteria for AGI, current LLMs hardly qualify as AGI.
- The record does not meet the current rules. Our reading, not assessed: As of 2023, computer vision is still far from a solved problem.
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: In many practical problems, working practitioners don't use transformer-based LLMs; they use a wide range of other methods that are often far more tractable, far more reliable, and far more efficient.
- The record does not meet the current rules. Our reading, not assessed: Current machines can't count or do five-digit multiplication, can't reliably generalize kinship relations or reliably follow the rules of chess, can't obey the dictum 'don't make stuff up' when writing a biography or a new story, and the authors have yet to see them drive cars as well as the average human adult.
- The two reviews read the claim differently