The new AI scaling law shell game
We recorded 12 claims from this post. 6 were checked against evidence. 6 can’t be checked. 4 more are held. Read the post at its source.
Claims from this post (12)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
Observation
“But things have been changing rapidly, and the great scaling retrenchment has already begun.”
Our reading: As of November 2024, a major retreat from scaling claims in the AI field has already begun.
No horizon Evidence as of
Limitation: Great retrenchment has no stated threshold.
Assessment: UnresolvedSaid
Observation
“The new version is that there is not one scaling law, but three: scaling with how long you train a model (which isn’t really holding anymore), scaling with how long you post-train a model, and scaling with how long you let a given model wrestle with a given problem (or what Satya Nadella called scaling with “inference time compute”).”
Our reading: Scaling performance by training models longer is no longer really holding.
No horizon Evidence as of
Limitation: The strongest experimental counterevidence concerns an earlier model and is context only.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“I suggest you keep your hands on your wallets here.”
Our reading: People should be financially wary of the new claims about multiple AI scaling laws.
No horizon
Assessment: UnresolvedSaid
Forecast
“Like the first generalization, the new ones will eventually run out.”
Our reading: The new scaling laws (post-training and inference-time) will eventually stop holding.
No deadline stated Evidence as of
Limitation: A narrower reading limited to particular scaling recipes would be easier to support.
Assessment: SupportedSaid
Observation
“GPT-4 was better than GPT-3 in almost every way; o1 is a far less uniform improvement.”
Our reading: GPT-4 was better than GPT-3 in almost every way, whereas o1 is a far less uniform improvement.
No horizon Evidence as of
Limitation: The result depends on reading the comparison qualitatively across capability categories.
Assessment: SupportedSaid
Observation
“(In some cases o1 doesn’t even do better than GPT-4).”
Our reading: In some cases o1 does not perform better than GPT-4.
No horizon Evidence as of
Limitation: The supported comparison is principally against GPT-4o, not specifically the original GPT-4.
Assessment: Can’t be checkedSaid
Observation
“My guess is o1’s test time scaling works best in closed domains where you can create a lot of reliable synthetic data (such as math problems), and not so well in open-ended domains.”
Our reading: Marcus guesses that o1's test-time scaling works best in closed domains where a lot of reliable synthetic data can be created, such as math problems, and not so well in open-ended domains.
No horizon
Qualifier: my guess
Assessment: UnresolvedSaid
Observation
“Even if you look at OpenAI’s own research post on o1, which introduced the newest putative law, it is immediately apparent that scaling test time compute works for some problems and not others.”
Our reading: OpenAI's own o1 research shows that scaling test-time compute helps on some problems but not others.
No horizon Evidence as of
Limitation: The conclusion depends on distinguishing model comparisons from compute-scaling experiments.
Assessment: Can’t be checkedSaid
Observation
“Third, scaling laws aren’t really laws anymore (if they ever were). The meaning of the word scaling has been greatly devalued.”
Our reading: Scaling laws aren't really laws anymore, if they ever were, and the meaning of the word 'scaling' has been greatly devalued.
No horizon
Qualifier: if they ever were
Assessment: Can’t be checkedSaid
Rhetoric
“Trying to rescue LLMs by making a system that is inherently blind to factuality run longer — rather than better—is like smearing water around wet plates.”
Our reading: Making LLMs, which are inherently blind to factuality, run longer rather than better is a futile way to rescue them.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“We can and must do better. Scaling can only take us so far; the time has come for fresh invention.”
Our reading: The AI field must move beyond scaling and pursue genuinely new ideas.
No horizon
Assessment: Can’t be checkedSaid
Observation
“Scaling can only take us so far; the time has come for fresh invention.”
Our reading: Scaling alone has limited ability to advance AI.
No horizon
Held from this post (4)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The record does not meet the current rules. Our reading, not assessed: In recent weeks, much of the AI field has quietly acknowledged that recent unreleased large models are not as powerful as scaling laws predicted.
- The record does not meet the current rules. Our reading, not assessed: The alleged scaling law Nadella emphasized, inspired by OpenAI's o1 (the more time spent on a problem, the more accurate the result), is expensive because it by definition burns up more GPU time, and it is unreliable.
- The two reviews reached different assessments. Our reading, not assessed: The new scaling laws, like the original, are empirical generalizations rather than physical laws.
- The two reviews read the claim differently