“Nobody has been able to beat it decisively. Places like Google and Anthropic have put in a lot of money trying. None succeeded; instead there tentatively seems to be convergence at GPT-4 levels.”
Our reading:
In GPT-4's first year, no competitor including Google and Anthropic decisively beat it; models appear to be converging at GPT-4 levels.
Assessment: Supported
What happened
Was it true for the period described?
Outcome: Supported
The evidence matches what they said.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
The cited comparisons support the claim of no decisive overall lead over the contemporary GPT-4 family through March 16, 2024 [e1–e3]. On March 7, LMSYS described Claude 3 Opus as rivaling GPT-4 Turbo in Arena's human-preference comparison [e1]. On March 8, Aider reported that Opus completed 68.4% of 133 Python exercises within two tries, only a few points above GPT-4 Turbo [e2]. Google's December 2023 report claimed Gemini Ultra benchmark advantages, including 90.04% on MMLU with a specialized prompting method; these developer-reported results do not establish a decisive overall lead [e3]. These records are consistent with the tentative suggestion of convergence at GPT-4-class performance during the period [e1–e3].
Caveats
- The assessment uses the contemporary GPT-4 family, including Turbo; it does not separately settle comparison with the original March 2023 model.
- Arena measures human preferences, and Aider tests a specific code-editing workflow. Neither covers every capability or competitor.
- Google's results are developer-reported and sensitive to evaluation methods.
- The independent comparisons precede March 16 by several days.
- The evidence establishes neither a permanent capability ceiling nor the absence of individual benchmark wins.
Evidence (3 items)
Supports the claim: LMSYS Chatbot Arena,
rival GPT-4-Turbo
Why it matters: The independent benchmark operator described rivalry and close matching, consistent with competitors reaching GPT-4-class performance. This measures human preference in the Arena, not every capability.
- Publisher
- LMSYS Chatbot Arena
- Kind of source
- Benchmark result
- About events on
- Published
Supports the claim: Aider / Paul Gauthier,
only a few points
Why it matters: Opus won this code-editing evaluation, which is material counterevidence to any claim that competitors never surpassed GPT-4. The evaluator nevertheless described a small margin, supporting the narrower claim that no decisive overall lead had been established.
- Publisher
- Aider / Paul Gauthier
- Kind of source
- Independent measurement
- About events on
- Published
Context only: Google Gemini Team / arXiv,
an accuracy of 90.04%
Why it matters: Google reported genuine benchmark advantages, so 'none succeeded' cannot fairly mean no benchmark wins. These developer-reported results, including a prompting-sensitive MMLU result, do not alone establish the clear overall advantage required by the contract.
- Publisher
- Google Gemini Team / arXiv
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Where we had to interpret
- The anniversary context describes GPT-4's first year, through the post date.
- The plain reading of 'decisively' is a clear overall capability advantage, rather than a win on an individual benchmark.
- I read GPT-4 as the contemporary GPT-4 family, including Turbo. Restricting it to the original March 2023 model could change the assessment.
- The convergence statement describes competitors approaching the same broad capability class. Its tentative wording does not assert a permanent ceiling.
- The reference to investment provides context for the capability comparison; this assessment does not grade an unspecified spending amount.
Right if: Contemporaneous comparisons show leading competitors approaching GPT-4-class performance, with any advantages limited, modest, or dependent on the task, rather than establishing a clear overall lead over the contemporary GPT-4 family.
Wrong if: A competitor demonstrated a substantial and consistent overall capability advantage over the contemporary GPT-4 family during the target period, supported by comparable evaluations across substantially different tasks.
We wrote this test after the outcome was known.
We found 8 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, Happy 1st Birthday, GPT-4, paragraph 2.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-29. The post may have been edited after 2024-03-16.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, Happy 1st Birthday, GPT-4
Nobody has been able to beat it decisively. Places like Google and Anthropic have put in a lot of money trying. None succeeded; instead there tentatively seems to be convergence at GPT-4 levels.
- Where
- paragraph 2
- Published
- 2024-03-16
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Supported. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
The final check failed: the write-up said more than the evidence showed.