“But between the disappointments of GPT-5 and a new study from METR that shows that LLMs do markedly better on coding benchmarks than in real-word practice, I think it is safe to say that LLMs won’t lead the way.”
Our reading, the sentence that was checked:
LLMs do markedly better on coding benchmarks than in real-world coding practice.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, OpenAI’s Waterloo? [with corrections], paragraph 34.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-openais-waterloo-b64a45-2-a2
We recorded 8 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-08-14. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- a new study from METR
- LLMs do markedly better on coding benchmarks
- than in real-word practice
Our interpretation
- I read the explicit reference to a new METR study as a comparison between impressive benchmark performance and substantially weaker practical usefulness in that study's coding setting.
- 'Markedly better' describes a meaningful qualitative mismatch; benchmark success rates and developer completion times are not interchangeable numerical measures.
- The target period is February–June 2025, the period investigated by the study identified through the explicit reference. This is a historical study claim, not a claim about October 2026 tools.
- This reading does not establish that the same gap holds across all developers, tasks or LLMs.
Counts as supported if: The cited METR research documents substantially weak or negative practical coding effects despite impressive coding benchmark results for contemporary AI systems.
Counts as contradicted if: The cited research finds practical coding benefits commensurate with benchmark promise, or its reported results otherwise directly reject the asserted mismatch.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Support applies to the finding in METR’s February–June 2025 coding study, not to a universal comparison of benchmarks with real-world coding [e1, e2]. In that trial, 16 experienced open-source developers worked on 246 real issues; allowing AI increased completion time by 19% [e1]. METR contrasted this slowdown with impressive benchmark scores [e1]. This supports a qualitative mismatch between benchmark performance and practical productivity in the studied setting [e1, e2].
Limitations
- The study does not establish the same mismatch across all developers, tasks or LLMs; it allows for substantial productivity gains in other coding settings [e2].
- Benchmark task success and developer completion time are different measures, so these results do not quantify a gap on a common scale [e1].
- The supplied passages do not identify the full set of models used in the trial [e1, e2].
- The follow-up begun in August 2025 showed indications of speedup, but METR’s February 24, 2026 update described selection and time-measurement problems that limited conclusions about later tools [e3].
Evidence (3 items)
Supports the claim: METR,
impressive benchmark scores
Why it matters here: The trial involved 16 developers and 246 real issues; allowing AI increased completion time by 19%. METR explicitly contrasts this practical slowdown with impressive benchmark performance, supporting the study-specific reading.
- Publisher
- METR
- Kind of source
- Independent measurement
- Describes events of
- Public from
Context only: METR / arXiv,
overgeneralizing
Why it matters here: The paper studies February–June 2025 tools and experienced developers working in familiar, mature repositories. It allows that other coding settings may see substantial gains, so it does not establish a universal benchmark-to-practice gap.
- Publisher
- METR / arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: METR,
evidence for speedup
Why it matters here: The later experiment suggests improved productivity but has selection and measurement problems. It concerns later tools and is context only; it does not overturn the historical finding.
- Publisher
- METR
- Kind of source
- Independent measurement
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, OpenAI’s Waterloo? [with corrections]
But between the disappointments of GPT-5 and a new study from METR that shows that LLMs do markedly better on coding benchmarks than in real-word practice, I think it is safe to say that LLMs won’t lead the way.
- Where
- paragraph 34
- Published
- 2025-08-14
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-openais-waterloo-b64a45-2-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.