What “game over” for the latest paradigm in AI might look like
We recorded 13 claims from this post. 6 were checked against evidence. 7 can’t be checked. 1 more is held. Read the post at its source.
Claims from this post (13)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
Causal claim
“His hypothesis, now generally called scaling maximalism, remains extremely popular, in no small part because bigger and bigger models have indeed continued to do ever more impressive things.”
Our reading: Scaling maximalism remains extremely popular, in no small part because bigger and bigger AI models have continued to do ever more impressive things.
No horizon Evidence as of
Limitation: The survey concerns NLP researchers, not all AI researchers, companies, investors, or the public.
Assessment: SupportedSaid
Observation
“Scaling is an empirical observation, not a guaranteed, continued law of nature.”
Our reading: AI scaling is an empirical observation, not a law of nature that is guaranteed to keep holding.
No horizon Evidence as of
Limitation: An empirical relationship can remain useful and predictive without being guaranteed.
Assessment: Can’t be checkedSaid
Conditional forecast
“The proof makes too many assumptions to be taken as gospel, but if it is even close to correct, there may soon be real trouble in Scaling City.”
Our reading: If the Merrill, Warstadt and Linzen proof is even close to correct, scaling maximalism may soon be in real trouble.
No horizon
Only if: if the Merrill/Warstadt/Linzen proof is even close to correct
Assessment: Can’t be checkedSaid
Forecast
“There might not be enough data in the world to make scaling maximalism work.”
Our reading: There may not be enough data in the world for scaling maximalism to work.
No horizon
Assessment: Can’t be checkedSaid
Forecast
“There might not be enough available compute in the world to make scaling maximalism feasible.”
Our reading: There may not be enough available compute in the world to make scaling maximalism feasible.
No horizon
Assessment: SupportedSaid
Observation
“Scaling here was largely AWOL; even the best model was only at 80.6%, and for most models, scaling had at best a neglible effect.”
Our reading: On the Ruis et al. pragmatics task, the best model reached only 80.6%, and for most models scaling had at best a negligible effect.
No horizon Evidence as of
Limitation: Support depends on reading 80.6% as the overall mean, not the maximum across every prompt configuration.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“If you get syntax and semantics but fail on pragmatics or commonsense reasoning, you don’t have AGI you can trust.”
Our reading: A system that gets syntax and semantics right but fails on pragmatics or commonsense reasoning is not AGI you can trust.
No horizon
Scope: system gets syntax and semantics but fails on pragmatics or commonsense reasoning
Assessment: Can’t be checkedSaid
Conditional forecast
“What hit me hard, as I was reading the paper, is that asymptotic 80% performance on even a single important task like this might spell game over for scaling.”
Our reading: If performance plateaus at about 80% on even one important task like pragmatics, that may mean game over for scaling.
No horizon
Only if: if performance plateaus at about 80% on a single important task
Assessment: UnresolvedSaid
Observation
“As I have long argued, capturing this without cognitive models and common sense is really hard.”
Our reading: Capturing the pragmatic implications of language without cognitive models and common sense is really hard.
No horizon Evidence as of
Limitation: The finding does not deny the demonstrated difficulty.
Assessment: Can’t be checkedSaid
Forecast
“Some important tasks might simply not scale.”
Our reading: Some important tasks may simply not improve with scale.
No horizon
Assessment: UnresolvedSaid
Forecast
“Scaling maximalism is an interesting hypothesis, but I stand by my prediction that it won’t get us to AGI.”
Our reading: Scaling maximalism will not get us to AGI.
No deadline stated Evidence as of
Limitation: This is not a finding that AGI is absent everywhere; it is a finding that the reviewed evidence does not settle this forecast.
Assessment: UnresolvedSaid
Conditional forecast
“Any one of them would mean we need a real paradigm shift, if we are to get to AGI.”
Our reading: If any one of the three failure modes (not enough data, not enough compute, tasks that don't scale) holds, reaching AGI will need a real paradigm shift.
No deadline stated Evidence as of
Only if: if any of the three failure modes (data, compute, non-scaling tasks) holds
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“All of this points to the same conclusion: if we want to reach AGI we shouldn’t keep putting so many eggs in the scaling-über-alles basket.”
Our reading: If the goal is AGI, the AI field should stop putting so much of its effort into scaling alone.
No horizon
Qualifier: if we want to reach AGI
Held from this post (1)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews reached different assessments. Our reading, not assessed: Gains on some measures over months or years do not show that scale is all AI needs.