“In the last 18 months, GPT-4o is not significantly different.”
Our reading:
GPT-4o is not significantly different from what was available 18 months earlier.
Assessment: Contradicted
What happened
Was it true for the period described?
Outcome: Contradicted
The evidence goes against what they said.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
The contract compares GPT-4o with the GPT-4 line OpenAI had offered since March 2023. The evidence shows clear differences. In an LMSYS data file dated 2024-07-23, GPT-4o scored a 79.21 Arena-Hard-Auto win rate against GPT-4-0314, the baseline model (e4). OpenAI's 2024-05-13 launch post says GPT-4o is a single end-to-end model across text, vision and audio. The same post puts its average audio response time at 320 ms, against 5.4 seconds for GPT-4 in the earlier Voice Mode (e3). It also says GPT-4o is 2x faster and half the price of GPT-4 Turbo in the API (e2). These differences meet the contract's contradiction condition. One narrower point holds. OpenAI said GPT-4o reached 'GPT-4 Turbo-level performance' on text, reasoning and coding benchmarks (e1), and it called GPT-4o 'a GPT-4 level model' (e2). That supports text parity with GPT-4 Turbo, but it does not show that GPT-4o was not significantly different from GPT-4 across the 18 months.
Caveats
- He set no threshold for 'significantly', so the result depends on the contract's ordinary reading.
- Arena-Hard-Auto uses a GPT-4-class model as judge, which may favour OpenAI models. Its win rates may not match real-world differences.
- The latency, speed, price and modality claims come from OpenAI's own launch post, not from independent measurement.
- The date of the LMSYS file comes from its own date column, not from a separate timestamp.
- The evidence covers only OpenAI's GPT-4 line. It does not address progress across the wider field.
Evidence (4 items)
Supports the claim: OpenAI,
GPT‑4o achieves GPT‑4 Turbo-level performance on text, reasoning, and coding intelligence
Why it matters: OpenAI itself says GPT-4o's text, reasoning and coding level matches the GPT-4 Turbo line. That supports the claim of no significant jump in core intelligence.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- About events on
- Published
Supports the claim: OpenAI,
As a first fruit of this research, we’re able to make a GPT‑4 level model available much more broadly.
Why it matters: OpenAI calls GPT-4o a 'GPT-4 level model'. The gain it describes is efficiency and access, not a higher level of capability.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- About events on
- Published
Goes against the claim: OpenAI,
Prior to GPT‑4o, you could use Voice Mode to talk to ChatGPT with latencies of 2.8 seconds (GPT‑3.5) and 5.4 seconds (GPT‑4) on average.
Why it matters: Elsewhere the same page says GPT-4o answers audio in about 320 ms on average, through a single end-to-end model. That cuts GPT-4's 5.4 s voice latency sharply and adds native audio and vision. These are substantial differences beyond text.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- About events on
- Published
Goes against the claim: LMSYS,
gpt-4o-2024-05-13,GPT-4o-2024-05-13,-,0.887,2023/10,Proprietary,OpenAI,https://openai.com/index/hello-gpt-4o/,79.21,77.5,80.94
Why it matters: In the same file, GPT-4-0314 is the Arena-Hard-Auto baseline at 50.0. GPT-4o scores 79.21, meaning it was judged better than the original March 2023 GPT-4 on most hard prompts. The file also lists MMLU at 0.887 versus 0.864 for GPT-4-0314.
- Publisher
- LMSYS
- Kind of source
- Benchmark result
- About events on
- Published
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Conditions
- in the last 18 months (their words)
Where we had to interpret
- Plain reading: by July 2024, OpenAI's GPT-4o (released May 2024) was not a significant step up from the GPT-4 line OpenAI had offered since March 2023. In other words, OpenAI's capability had stalled since GPT-4.
- The baseline is GPT-4 (March 2023) and its GPT-4 Turbo updates, not pre-GPT-4 models. That matches his point about stalling and avoids counting GPT-4's own launch against him.
- He set no threshold for 'significantly' and named no domain. Read in context ('what if they've stalled'), he means general capability, chiefly text intelligence. Speed, price and new modes (voice, vision) are also relevant, because they are what OpenAI itself presented as the differences.
Right if: Primary or benchmark evidence shows GPT-4o performing at about the same level as GPT-4 / GPT-4 Turbo, with no large gain in capability.
Wrong if: Primary or benchmark evidence shows GPT-4o clearly better than GPT-4 on core capability, or adding major new capabilities, in a way an informed reader would call significant.
We wrote this test after the outcome was known.
We found 9 claims in this post. See them all.
Full record
Source and dates
Where's Your Ed At, Rot Economics - An Interview With MIT's Daron Acemoglu, paragraph 53.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-28. The post may have been edited after 2024-07-24.
Where it was said (1)
Said once in the posts we read.
: Where's Your Ed At, Rot Economics - An Interview With MIT's Daron Acemoglu
In the last 18 months, GPT-4o is not significantly different.
- Where
- paragraph 53
- Published
- 2024-07-24
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Contradicted. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
A reviewer said our test wasn’t fair to what was actually said.