An AI rumor you won’t want to miss
We recorded 6 claims from this post. 3 were checked against evidence. 3 can’t be checked. Read the post at its source.
Claims from this post (6)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
“Crucially, there has never been any principled argument that they would solve hallucinations or reasoning, or edge cases in open-ended worlds —or that synthetic data would suffice indefinitely to keep feeding the machine.”
What we tested: There has never been a principled argument that scaling would solve hallucinations, reasoning, or open-world edge cases, or that synthetic data would suffice indefinitely.
Assessment: SupportedSaid
“They are merely empirical generalizations that held for a certain period time, when there was enough fresh data and compute.”
What we tested: Scaling laws are merely empirical generalizations that held for a limited period while enough fresh data and compute were available.
What happened: The authors explicitly characterize their loss relationships as empirical. Their reported measurements cover finite ranges of model size, data and training compute. (arXiv; Kaplan et al., )
Assessment: Can’t be checkedSaid
“As I wrote yesterday (and first noted back in April), I strongly suspect that, contra Altman, we have in fact reached a point of diminishing returns for pure scaling.”
What we tested: Marcus strongly suspects that pure scaling of AI models has reached a point of diminishing returns, as of November 2024.
Assessment: Can’t be checkedSaid
“Just adding data and compute and training longer worked miracles for a while, but those days may well be over.”
What we tested: The era when simply adding data and compute produced large gains may well be over.
Assessment: Can’t be checkedSaid
“If the rumor is verified, there could be the AI equivalent of a bank run.”
What we tested: If the rumor that a second major AI company is seeing diminishing returns from scaling is verified, the AI industry could see the equivalent of a bank run.
Assessment: SupportedSaid
“Scaling was always a hypothesis.”
What we tested: Scaling (improving AI by adding more data and compute) was always a hypothesis.
What happened: Early scaling research built its claims from empirical measurements across several tasks. This supports the distinction between observed regularities and guarantees about their indefinite continuation. (arXiv; Hestness et al., )