Deep Learning, Deep Scandal
We recorded 8 claims from this post. 4 were checked against evidence. 4 can’t be checked. 1 more is held. Read the post at its source.
Claims from this post (8)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Observation
“OpenAI failed at this (calling their best result GPT-4.5, and recently announcing a further delay on GPT-5); Grok failed at this (Grok 3 is no GPT 5). Google has failed at reaching “GPT-5” level, Anthropic has, too.”
Our reading: As of the post's publication on April 7, 2025, OpenAI, xAI (Grok), Google and Anthropic had each failed to reach what Marcus calls "GPT-5 level" AI.
No horizon
Qualifier: as of April 7, 2025
Assessment: Can’t be checkedSaid
Observation
“One of the clearest signs of this is the saga of the just-released Llama 4, the latest failed billion (?) dollar attempt by one of the majors to create what we might call GPT-5 level AI.”
Our reading: Meta's Llama 4 is a failed attempt to create GPT-5 level AI.
No horizon Evidence as of
Limitation: This assessment does not establish whether Llama 4 met internal targets or expectations.
Assessment: UnresolvedSaid
Observation
“Deep learning is indeed finally hitting a wall, in the sense of reaching a point of diminishing results.”
Our reading: Deep learning is hitting a wall in the sense of reaching diminishing returns.
No horizon Evidence as of
Limitation: METR evaluated an earlier GPT-4.5 checkpoint in a short, preliminary evaluation.
Assessment: SupportedSaid
Observation
“First, the AI community is in fact pretty disappointed with LLama 4’s performance:”
Our reading: The AI community is disappointed with Llama 4's performance.
No horizon Evidence as of
Limitation: Public commentary is self-selected.
Assessment: ContradictedSaid
Observation
“If the juicing of benchmarks and “blending” of test data is the primary scandal, there is a secondary scandal too: a bunch of well-paid pundits, such as Roose, Newton, and Cowen, are failing to reckon with the repeated failures of massive companies to build GPT-5.”
Our reading: A number of well-paid pundits, such as Roose, Newton and Cowen, are failing to reckon with the repeated failures of massive companies to build GPT-5, which Marcus calls a secondary scandal next to benchmark juicing.
No horizon Evidence as of
Limitation: The result depends on reading reckoning as substantive consideration, rather than agreement about how much weight setbacks deserve.
Assessment: Can’t be checkedSaid
Rhetoric
“Just because Meta or OpenAI wants you to believe something doesn’t mean it is true.”
Our reading: Meta or OpenAI wanting people to believe something does not by itself make it true.
No horizon
Assessment: UnresolvedSaid
Observation
“Real world customers, many of whom are disappointed, are rarely mentioned.”
Our reading: Real-world customers, many of whom are disappointed, are rarely mentioned in the pundits' discussion of AI progress.
No horizon Evidence as of
Limitation: The survey was commissioned by an AI vendor and measures adoption disappointment, which can include organizational problems as well as product shortcomings.
Assessment: Can’t be checkedSaid
Observation
“The reality, reported or otherwise, is that large language models are no longer living up to expectations, and its purveyors appear to be making dodgy choices to keep that fact from becoming obvious.”
Our reading: LLMs are no longer living up to expectations, and their vendors appear to be making dodgy choices to conceal this.
No horizon
Held from this post (1)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The record does not meet the current rules. Our reading, not assessed: Model performance cannot be predicted entirely from model and data size; Meta wasted massive sums on that mistaken hypothesis.