“Suleyman’s colleague Kevin Scott hinted that GPT-5 would blow away PhDs, and be vastly better than GPT-4; in reality the improvements were far more modest.)”
Our reading:
GPT-5's improvements over GPT-4 were far more modest than Kevin Scott had hinted.
Assessment: Can’t be checked
What happened
Not graded on evidence
Outcome: Can’t be checked
An opinion, a recommendation, or too vague to test. Listed, not scored.
Evidence up to . Checked .
Why
On June 5, 2024, The Decoder reported that Kevin Scott said early looks at next-generation models suggested they could pass PhD qualifying exams. He said GPT-4-class models perform about as well as high school students on AP exams (e1). Scott added that this 'probably sounds like a bigger deal than it actually is' (e1). The report does not show him naming GPT-5. 'Blow away PhDs' is Marcus's paraphrase, not Scott's wording. On August 7, 2025, OpenAI said GPT-5 pro scored 88.4% on GPQA without tools (e2). OpenAI reported this figure itself, and the evidence gives no GPT-4 score to compare it with. The claim gives no measure for 'far more modest', and the hint it compares against is hedged and does not name a model. So the claim cannot be scored as true or false.
Caveats
- The GPQA figure comes from OpenAI and was not independently checked.
- The figure is for GPT-5 pro with extended reasoning, not necessarily for the standard GPT-5 model.
- Scott's remarks are known only from a news report (e1), not from a recording or transcript in the evidence.
- The evidence gives no GPT-4 score on the same benchmarks, so it does not show how big the improvement was.
- The evidence does not compare GPT-5's real-world usefulness with GPT-4's.
Evidence (2 items)
Claims like this aren’t scored, so there is no evidence to show.
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Where we had to interpret
- Scott's recorded June 2024 remark was that next-generation models 'could pass PhD qualifying exams', compared with GPT-4 matching high schoolers on AP exams. He also said it 'probably sounds like a bigger deal than it actually is'. He did not name GPT-5 in the coverage I read.
- 'Blow away PhDs' is Marcus's paraphrase. 'Far more modest' has no measure. On benchmarks such as GPQA, GPT-5 scores very high, but whether the real-world gain was 'far more modest' than a hint is a judgment call.
- There is no plain, agreed measure for comparing a hint to an outcome, so I treat this as not adjudicable.
We don’t grade claims like this. We only check the quote and who said it.
We found 9 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, Promises are cheap, paragraph 8.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-28. The post may have been edited after 2026-02-12.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, Promises are cheap
Suleyman’s colleague Kevin Scott hinted that GPT-5 would blow away PhDs, and be vastly better than GPT-4; in reality the improvements were far more modest.)
- Where
- paragraph 8
- Published
- 2026-02-12
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Can’t be checked. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
The final check failed: the write-up said more than the evidence showed.