“GPT-4 was better than GPT-3 in almost every way; o1 is a far less uniform improvement.”
Our reading, the sentence that was checked:
GPT-4 was better than GPT-3 in almost every way, whereas o1 is a far less uniform improvement.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, The new AI scaling law shell game, paragraph 12.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-a-new-ai-scaling-law-shell-game-264969-a1
We recorded 12 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-11-24. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- in almost every way (qualifier, stated by the speaker)
From the speaker
- GPT-4 was better than GPT-3
- in almost every way
- o1 is a far less uniform improvement
Our interpretation
- Read almost every way as broad task capability, not every individual prompt, price, latency or product feature.
- Keep GPT-3 distinct from GPT-3.5.
- Read the o1 comparison against the GPT-4 family, principally GPT-4o, which is the comparator in the research Marcus discusses.
- Assess the original GPT-3-to-GPT-4 comparison and the o1 generation represented in the September–November 2024 public record. No common beginning date is specified.
- Read far less uniform qualitatively as broad gains in the older comparison versus clear task-dependent strengths and non-improvements in the newer comparison.
Counts as supported if: Broad evaluations show gains across the main tested capability categories from GPT-3 to GPT-4, while contemporary o1 evaluations identify substantial strengths alongside task categories where it is not preferred to GPT-4o.
Counts as contradicted if: Broad evaluations show that GPT-4's gains over GPT-3 were similarly patchy, or that contemporary o1 improved broadly without the alleged category-level exceptions.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The qualitative comparison is supported. GPT-Fathom directly documents broad GPT-3-to-GPT-4 capability gains. OpenAI's o1 evaluation reports strong reasoning gains alongside natural-language categories where o1-preview was not preferred to GPT-4o. That supports less uniform improvement on the ordinary capability reading. It does not establish a numerical difference in uniformity or superiority on every possible task.
Limitations
- The result depends on reading the comparison qualitatively across capability categories.
- The older and newer studies use different benchmark collections.
- Human preference, benchmark accuracy and product usefulness are related but distinct measures.
- The findings concern the contemporary models, not later o1 releases.
Evidence (3 items)
Supports the claim: Zheng et al., ByteDance and University of Illinois Urbana-Champaign, arXiv,
most benchmarks.
Why it matters here: The study evaluates legacy GPT-3 models and GPT-4 under aligned settings across more than twenty benchmarks and seven capability categories. Its overall analysis reports broad improvements across those categories from GPT-3 to GPT-4.
- Publisher
- Zheng et al., ByteDance and University of Illinois Urbana-Champaign, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: OpenAI,
not preferred
Why it matters here: OpenAI's comparison finds strong preferences for o1-preview on reasoning-heavy tasks but exceptions on natural-language tasks. This directly supports the claimed unevenness, while the same publication documents substantial reasoning gains.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: ARC Prize,
both outperform GPT-4o
Why it matters here: Independent testing found improvements on an abstract-reasoning benchmark as well. This limits any reading that o1's benefits were confined to the particular math and coding examples, but does not establish uniform gains.
- Publisher
- ARC Prize
- Kind of source
- Independent measurement
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, The new AI scaling law shell game
GPT-4 was better than GPT-3 in almost every way; o1 is a far less uniform improvement.
- Where
- paragraph 12
- Published
- 2024-11-24
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-a-new-ai-scaling-law-shell-game-264969-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-a-new-ai-scaling-law-shell-game-264969-a1.