“Huge jump from GPT-2 to GPT-3. Huge jump from GPT-3 to GPT-4 … and not so much for GPT-4 (13 months ago) to GPT-4 Turbo (just released). It’s hard not to see this plot as tentative evidence for the hypothesis of diminishing returns.”
Our reading, the sentence that was checked:
MMLU gains from GPT-4 to GPT-4 Turbo are small compared with earlier generational jumps, tentatively evidencing diminishing returns.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Evidence that LLMs are reaching a point of diminishing returns - and what that might mean, paragraph 11.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-evidence-that-llms-are-reaching-a-f2cb8e-a2
We recorded 20 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-04-13. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- tentative evidence (qualifier, stated by the speaker)
From the speaker
- Huge jump from GPT-2 to GPT-3
- Huge jump from GPT-3 to GPT-4
- not so much for GPT-4 (13 months ago) to GPT-4 Turbo (just released)
- tentative evidence for the hypothesis of diminishing returns
- y- axis is percent accuracy
Our interpretation
- Compare absolute percentage-point gains on MMLU across the named models, using the largest GPT-3 model and the April 2024 GPT-4 Turbo snapshot.
- The ordinary reading is that the last reported gain is substantially smaller than both earlier gains. No precise threshold for 'huge' or 'not so much' is supplied.
- Assess the historical score pattern and its tentative evidential value. This is not a prediction that future models will stop improving or a demonstrated relationship between training expenditure and capability.
- The target covers the named historical generations through the post date. Its start is unspecified because the passage gives no starting calendar date.
- Evaluation methods must be examined. Differences limit the strength of the inference even when the reported percentage-accuracy pattern is clear.
Counts as supported if: Published MMLU results show a substantially smaller GPT-4-to-April-2024-Turbo accuracy gain than both earlier generational gains, with methodological qualifications consistent with describing the pattern as tentative evidence.
Counts as contradicted if: Comparable published results show that the GPT-4-to-April-2024-Turbo gain is similar to or larger than either earlier generational gain, undermining the stated historical pattern.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The January 2021 MMLU paper reports 32.4% accuracy for fine-tuned GPT-2 and 43.9% for GPT-3 X-Large, a gain of 11.5 percentage points [e1]. OpenAI's March 2023 report gives GPT-4 a score of 86.4%, a further gain of 42.5 points [e1, e2]. OpenAI's April 11, 2024 evaluation record reports 86.5–86.7% for the April 9 Turbo snapshot, only 0.1–0.3 points above the original GPT-4 result [e2, e3]. These reported gains support the historical pattern of a substantially smaller final increase, but differing evaluation methods limit its value as evidence of diminishing returns [e1, e2, e3].
Limitations
- The records do not compare all four models under one protocol: GPT-2 was fine-tuned, GPT-3 used few-shot prompting, GPT-4 used five-shot prompting, and Turbo used zero-shot chain-of-thought prompting [e1, e2, e3].
- These MMLU results alone do not establish diminishing returns to training resources, a general slowdown across capabilities, or limits on future improvement [e1, e2, e3].
Evidence (3 items)
Supports the claim: Dan Hendrycks and coauthors, arXiv,
GPT-2 on UnifiedQA training data and our dev+val set.
Why it matters here: Table 1 reports 32.4% for fine-tuned GPT-2 and 43.9% for GPT-3 X-Large evaluated few-shot, a reported increase of 11.5 percentage points. The different evaluation methods prevent treating this as a controlled comparison of unchanged model preparation.
- Publisher
- Dan Hendrycks and coauthors, arXiv
- Kind of source
- Preprint, not peer reviewed
- Arithmetic
- 43.9 - 32.4 = 11.5 percentage points (inputs: Table 1: GPT-2 overall accuracy = 32.4%; Table 1: GPT-3 X-Large few-shot overall accuracy = 43.9%; One decimal place, matching the published scores.)
- Describes events of
- Public from
Supports the claim: OpenAI,
GPT-4’s performance along with the few-shot method used to evaluate.
Why it matters here: Table 2 reports GPT-4 MMLU accuracy of 86.4% with five-shot evaluation. Compared with the GPT-3 result in e1, the published increase is 42.5 percentage points.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Arithmetic
- 86.4 - 43.9 = 42.5 percentage points (inputs: e1: GPT-3 X-Large MMLU accuracy = 43.9%; Table 2: GPT-4 five-shot MMLU accuracy = 86.4%; One decimal place, matching the published scores.)
- Describes events of
- Public from
Supports the claim: OpenAI,
Evals are sensitive to prompting
Why it matters here: The initial release reports April Turbo MMLU scores of 86.5% with the chatgpt prompt and 86.7% with the assistant prompt, only 0.1–0.3 points above e2. It describes zero-shot chain-of-thought evaluation and warns about prompting sensitivity, limiting comparison with GPT-4's five-shot result.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Arithmetic
- 86.5 - 86.4 = 0.1; 86.7 - 86.4 = 0.3 percentage points (inputs: e2: original GPT-4 MMLU accuracy = 86.4%; April Turbo, chatgpt prompt: MMLU = 86.5%; April Turbo, assistant prompt: MMLU = 86.7%; One decimal place, matching the published scores.)
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Huge jump from GPT-2 to GPT-3. Huge jump from GPT-3 to GPT-4 … and not so much for GPT-4 (13 months ago) to GPT-4 Turbo (just released). It’s hard not to see this plot as tentative evidence for the hypothesis of diminishing returns.
- Where
- paragraph 11
- Published
- 2024-04-13
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-evidence-that-llms-are-reaching-a-f2cb8e-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-evidence-that-llms-are-reaching-a-f2cb8e-a2.