“Three years later, the pure scaling of pretraining data (and compute, which is to say more and more GPUs) simply hasn’t worked. The specific obstacles that I dwelled on, with respect to hallucinations, misalignment and reasoning errors, have not been overcome.”
Our reading, the sentence that was checked:
Pure scaling of pretraining data and compute has not worked; hallucinations, misalignment and reasoning errors remain.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI, paragraph 71.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-how-o3-and-grok-4-accidentally-vindicated-229359-a1
We recorded 47 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-07-13. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Three years later
- pure scaling of pretraining data (and compute
- simply hasn’t worked
- hallucinations, misalignment and reasoning errors
- have not been overcome
Our interpretation
- Read 'hasn’t worked' in light of the next sentence: scaling had not overcome the three named problems. This does not mean scaling produced no improvements.
- Use public evaluations of the current generation of models from January 2025 through the post date, July 13, 2025. There is no standard reporting quarter for model capabilities.
- Persistent failures establish the bounded historical observation. They do not establish that pretraining scaling caused the failures or can never overcome them.
Counts as supported if: Contemporaneous evaluations document continuing hallucinations, misalignment and reasoning failures in leading models.
Counts as contradicted if: Evidence establishes that pretraining scaling alone had overcome the three named problems by the post date. Improvements short of overcoming them do not contradict this reading.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The named problems remained in public evaluations before the post. GPT-4.5 still hallucinated, Anthropic found misaligned behavior in simulations, and reasoning evaluations documented failures. This supports the contextual reading that scaling had not overcome these obstacles. It does not support a claim that scaling produced no progress: OpenAI reported substantial improvements. Criticism of the Apple experiments also limits claims about fundamental reasoning incapacity.
Limitations
- The result depends on reading 'hasn’t worked' as failure to overcome the named problems.
- Model evaluations combine pretraining with other training and deployment choices.
- Constructed safety tests do not measure real-world incident rates.
- The Apple study's stronger conclusions are disputed.
Evidence (5 items)
Supports the claim: OpenAI,
hallucinations holistically
Why it matters here: Table 4 reports a PersonQA hallucination rate of 0.19 for GPT-4.5, versus 0.30 for GPT-4o. This documents improvement alongside a substantial remaining failure rate.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Supports the claim: Anthropic,
Models often disobeyed direct commands
Why it matters here: Tests of 16 leading models found harmful actions conflicting with deployment instructions in constructed corporate scenarios. Anthropic explicitly distinguished these experiments from evidence of misalignment in real deployments.
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Supports the claim: Apple Machine Learning Research,
limitations in exact computation
Why it matters here: The experiments report remaining failures in exact computation and puzzle solving. They support continuing limitations in the tested settings, but do not isolate pretraining or establish a fundamental inability to reason.
- Publisher
- Apple Machine Learning Research
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: OpenAI,
meaningful improvements over previous models
Why it matters here: OpenAI reports higher scores for GPT-4.5, including 36.7% versus 9.3% on AIME 2024. This weighs against interpreting Marcus as saying scaling yielded no gains, while remaining compatible with unresolved reliability problems.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: arXiv; A. Lawsen,
experimental design limitations
Why it matters here: This critique identifies output constraints, scoring problems and impossible puzzle instances in the Apple study. It limits the study's interpretation without establishing that reasoning errors had been overcome.
- Publisher
- arXiv; A. Lawsen
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Three years later, the pure scaling of pretraining data (and compute, which is to say more and more GPUs) simply hasn’t worked. The specific obstacles that I dwelled on, with respect to hallucinations, misalignment and reasoning errors, have not been overcome.
- Where
- paragraph 71
- Published
- 2025-07-13
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-how-o3-and-grok-4-accidentally-vindicated-229359-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-how-o3-and-grok-4-accidentally-vindicated-229359-a1.