“The difference between GPT-3 and GPT-4 was by all accounts massive; the difference between GPT-4 and o1 is by all accounts modest.”
Our reading, the sentence that was checked:
By all accounts the difference between GPT-3 and GPT-4 was massive, while by all accounts the difference between GPT-4 and o1 is modest.
Assessment: Contradicted Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Is OpenAI more like WeWork or Theranos?, paragraph 13.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-is-openai-more-like-wework-or-theranos-d8b385-a2
We recorded 11 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-10-20. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- by all accounts (qualifier, stated by the speaker)
From the speaker
- The difference between GPT-3 and GPT-4
- by all accounts massive
- the difference between GPT-4 and o1
- by all accounts modest
Our interpretation
- The ordinary reading asserts broad agreement that the earlier transition was transformative while the later transition offered only limited improvement.
- By all accounts is treated as a claim of broad agreement, not a requirement to inspect every informal opinion.
- For the later transition, assess o1-preview available at publication. GPT-4o comparisons inform the GPT-4 family comparison, with that substitution disclosed.
- Substantial documented gains across important reasoning domains can contradict the asserted agreement without proving that o1's overall improvement exceeded GPT-3 to GPT-4.
- The period covers the model generations discussed through publication; later o1 releases are excluded.
Counts as supported if: Broad contemporary evaluations agree on a large earlier improvement and only modest later improvement across meaningful capabilities.
Counts as contradicted if: Credible contemporary evaluations show substantial later gains across important capability domains, undermining the asserted agreement that the later improvement was merely modest.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Contradicted
The dated evidence goes against what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Contemporary evaluations undermine the asserted broad agreement that the later improvement was only modest [e1, e2]. On September 12, 2024, OpenAI reported that human trainers preferred o1-preview to GPT-4o by a large margin on reasoning-heavy data analysis, coding, and mathematics prompts in its company-run evaluation [e1]. An independent September 2024 mathematics exam evaluation reported o1-preview scores of 76 and 73 out of 76, compared with GPT-4o scores of 66 and 61 [e2]. These results challenge the claim about agreement without establishing an overall capability-distance comparison [e1, e2]. METR's testing through September 9 found tool-use and feedback-response difficulties with basic agent scaffolds, qualifying how broadly the reasoning results can be applied [e3].
Limitations
- The later comparisons use GPT-4o and o1-preview, rather than original GPT-4 and a later o1 release [e1–e3].
- OpenAI's preference evaluation does not establish improvement across all capabilities; it also found no preference for o1-preview on some natural-language tasks [e1].
- The independent mathematics results cover two runs on one examination [e2].
- The evidence does not establish the relative aggregate size of the two transitions. The earlier report's cited quantitative comparison uses GPT-3.5, leaving the exact GPT-3-to-GPT-4 comparison unsettled [e1–e4].
Evidence (4 items)
Goes against the claim: OpenAI,
a large margin
Why it matters here: The company reported substantial preference gains across three important reasoning domains for the actual preview model. This challenges the asserted agreement about only modest improvement, while remaining a company-run evaluation.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Goes against the claim: arXiv; Joost de Winter, Dimitra Dodou, and Yke Bauke Eisma,
76 and 73
Why it matters here: The same abstract reports GPT-4o scores of 66 and 61 out of 76. This independent example corroborates a meaningful mathematics improvement, although one examination cannot establish the size of an overall capability jump.
- Publisher
- arXiv; Joost de Winter, Dimitra Dodou, and Yke Bauke Eisma
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: METR,
struggle
Why it matters here: Independent testing found practical limitations in autonomous use. This supports caution about generalizing reasoning gains into an across-the-board improvement.
- Publisher
- METR
- Kind of source
- Independent measurement
- Describes events of
- Public from
Context only: arXiv; OpenAI,
substantially improves
Why it matters here: The report supports the general characterization of substantial earlier progress. Its surrounding quantitative comparison uses GPT-3.5, so it does not directly settle Marcus's GPT-3 comparison.
- Publisher
- arXiv; OpenAI
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Is OpenAI more like WeWork or Theranos?
The difference between GPT-3 and GPT-4 was by all accounts massive; the difference between GPT-4 and o1 is by all accounts modest.
- Where
- paragraph 13
- Published
- 2024-10-20
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-is-openai-more-like-wework-or-theranos-d8b385-a2: Current. Outcome: Contradicted. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-is-openai-more-like-wework-or-theranos-d8b385-a2.