“I argued that scaling as it was then defined (pretraining data and compute) alone was insufficient to solve challenges with reasoning, misalignment and hallucinations. (Spoiler alert: it still hasn’t.)”
Our reading, the sentence that was checked:
Scaling pretraining data and compute alone has not solved reasoning, misalignment and hallucination problems.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI, paragraph 63.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-how-o3-and-grok-4-accidentally-vindicated-b47ab1-a1
We recorded 36 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-07-13. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- scaling of pretraining data and compute alone (qualifier, stated by the speaker)
From the speaker
- scaling as it was then defined
- pretraining data and compute
- alone
- reasoning, misalignment and hallucinations
- it still hasn’t
Our interpretation
- Assess the present-status assertion that pretraining scale alone had not solved the three named problems, rather than whether Marcus previously made the argument.
- Solved means reliable handling across relevant tasks, not merely better average benchmark scores.
- This does not claim that future pretraining scale can never solve these problems.
- The word alone requires evidence distinguishing pretraining from post-training, tools, and other interventions.
- Use current-year evaluations through the post date, since technical evaluations have no common quarterly reporting cycle.
Counts as supported if: Evaluations specifically identifying the effects or outputs of pretraining scaling show unresolved reasoning, alignment, and hallucination problems during the target period.
Counts as contradicted if: A system improved through pretraining data and compute alone demonstrates reliable resolution of the named problems on representative evaluations.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The evidence is consistent with continued reasoning, safety, and hallucination problems, and it also documents meaningful gains from scaling. However, the reviewed frontier models combine pretraining with architecture changes, supervised fine-tuning, reinforcement learning, or tools. Their results do not isolate the pretraining-only assertion across all three named problems. Under the stated criteria, that evidence gap prevents a supported or contradicted outcome.
Limitations
- This is an evidence limitation, not evidence that pretraining alone had solved the problems.
- A looser assessment of deployed-model reliability would have stronger support, but would omit the explicit word alone.
- Current failures would not establish a permanent theoretical limit on scaling.
Evidence (2 items)
Context only: OpenAI,
shows meaningful improvements
Why it matters here: OpenAI reports reasoning improvements but remaining errors: GPT-4.5 scores 36.7% on AIME 2024 and 71.4% on GPQA. The same announcement describes architecture, optimization, and post-training changes, so the released model is not a clean pretraining-only experiment.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: OpenAI,
More work is needed
Why it matters here: The card reports a 19% PersonQA hallucination rate and imperfect jailbreak resistance. It also states that GPT-4.5 received supervised fine-tuning and reinforcement learning from human feedback, preventing these results from isolating pretraining scale alone.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
What would settle it
Dated base-model evaluations or controlled scaling studies covering reasoning, alignment, and hallucinations, with training details sufficient to isolate increased pretraining data and compute.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
I argued that scaling as it was then defined (pretraining data and compute) alone was insufficient to solve challenges with reasoning, misalignment and hallucinations. (Spoiler alert: it still hasn’t.)
- Where
- paragraph 63
- Published
- 2025-07-13
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-how-o3-and-grok-4-accidentally-vindicated-b47ab1-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-how-o3-and-grok-4-accidentally-vindicated-b47ab1-a1.