“The new version is that there is not one scaling law, but three: scaling with how long you train a model (which isn’t really holding anymore), scaling with how long you post-train a model, and scaling with how long you let a given model wrestle with a given problem (or what Satya Nadella called scaling with “inference time compute”).”
Our reading, the sentence that was checked:
Scaling performance by training models longer is no longer really holding.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, The new AI scaling law shell game, paragraph 6.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-a-new-ai-scaling-law-shell-game-69084e-a2
We recorded 12 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-11-24. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- scaling with how long you train a model
- which isn’t really holding anymore
Our interpretation
- Assess the parenthetical claim about pretraining, rather than the separate enumeration of three scaling approaches.
- The plain reading is that increasing pretraining scale no longer reliably delivers the capability gains previously expected. It need not mean that every additional training step produces no benefit.
- The surrounding last-few-weeks discussion places the current state in November 1–24, 2024. Older experiments provide context only.
- Training duration is read in the surrounding context of pretraining compute, model size and data, rather than wall-clock time alone.
Counts as supported if: Comparable predictions and measured frontier pretraining results for the claimed period show systematic shortfalls in expected capability gains.
Counts as contradicted if: Comparable predictions and measured results for that period show that increasing pretraining scale continued reliably to deliver the expected gains.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The available experimental record includes an earlier successful capability prediction for Llama 3. It does not establish what happened in the November runs Marcus discusses. Without comparable predictions and measured outcomes for that period, neither disappointing expectations nor earlier successes settle whether pretraining scaling had stopped reliably delivering expected gains.
Limitations
- The strongest experimental counterevidence concerns an earlier model and is context only.
- Private frontier training results and their original predictions were not available in the retrieved records.
- Results on one benchmark would not establish reliability across all capabilities.
Evidence (2 items)
Context only: Meta researchers via arXiv,
We show the results of this experiment on the ARC Challenge benchmark in Figure 4).
Why it matters here: The accompanying results report an accurate extrapolation across four orders of magnitude, slightly underpredicting Llama 3's final performance. This is relevant counterevidence from an earlier model, but cannot directly settle a claim about a subsequent November breakdown.
- Publisher
- Meta researchers via arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: DeepMind researchers via arXiv,
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget.
Why it matters here: The research treats training compute, model size and token count jointly. It explains why an unspecified claim about training longer needs a defined comparison, but provides no November 2024 outcome.
- Publisher
- DeepMind researchers via arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Contemporaneous predictions and measured outcomes from multiple relevant pretraining runs, with consistent metrics and enough detail to distinguish ordinary diminishing returns from prediction failure.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, The new AI scaling law shell game
The new version is that there is not one scaling law, but three: scaling with how long you train a model (which isn’t really holding anymore), scaling with how long you post-train a model, and scaling with how long you let a given model wrestle with a given problem (or what Satya Nadella called scaling with “inference time compute”).
- Where
- paragraph 6
- Published
- 2024-11-24
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-a-new-ai-scaling-law-shell-game-69084e-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-a-new-ai-scaling-law-shell-game-69084e-a2.