“METR’s blog and its tweets (perhaps written by publicists or by generative AI rather than by scientists), however, dropped the careful qualifications and nuance of the technical paper and made claims that go far beyond anything that the study actually supports.”
Our reading, the sentence that was checked:
METR's blog and tweets made claims far beyond what its study supports, dropping the technical paper's qualifications.
Assessment: Mixed Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, The latest AI scaling graph - and why it hardly makes sense, paragraph 8.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-the-latest-ai-scaling-graph-and-why-f680c2-a1
We recorded 5 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-05-03. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- METR’s blog and its tweets
- dropped the careful qualifications and nuance
- made claims that go far beyond anything that the study actually supports
Our interpretation
- Assess the March launch and April update discussed in the post.
- Compare domain restrictions, success probabilities, extrapolation conditions, and uncertainty statements across the paper and public communications.
- Distinguish omissions in an opening tweet from omissions throughout the blog or complete thread.
- The speculation about publicists or generative AI is hedged and is not assessed.
Counts as supported if: The communications materially omit the study’s restrictions and present conclusions extending beyond its tested scope or conditional findings.
Counts as contradicted if: The communications retain material restrictions and uncertainty, and present broader conclusions as conditional extrapolations.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Mixed
Part of the claim holds and part does not, and the record says which part is which.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The opening March tweet omits the domain and success-rate qualifications, supporting Marcus’s narrower concern about presentation. But the contemporaneous blog reproduction retains substantial caveats, and the April announcement explicitly refers to METR’s own tasks. The general description of the blog and tweets as having dropped the qualifications is therefore too broad. The stronger assertion that their claims went far beyond the study is not established merely by the headline omissions.
Limitations
- This assesses the identified announcements, not every METR tweet.
- Historical blog wording is corroborated through a contemporaneous reproduction rather than an immutable archive.
- The mixed judgment depends on distinguishing partial omission from wholesale removal.
Evidence (3 items)
Supports the claim: METR,
When will AI systems be able to carry out long projects independently?
Why it matters here: The accompanying text announces a roughly seven-month doubling of task length without stating the software-task restriction or 50% success threshold. This supports the narrower complaint about qualifications omitted from the opening tweet, rather than establishing that an entire thread lacked them.
- Publisher
- METR
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Goes against the claim: LessWrong, reproduction by Zach Stein-Perlman,
However, there remains the possibility of substantial model error.
Why it matters here: The dated reproduction preserves this warning, methodological dependence, conditional extrapolation, and discussion of external-validity uncertainty. It contradicts a general claim that the blog discarded the paper’s qualifications.
- Publisher
- LessWrong, reproduction by Zach Stein-Perlman
- Kind of source
- Secondary summary
- Describes events of
- Public from
Goes against the claim: METR,
For each model, we examined how capable it is on our tasks & how often it tries to “hack” them.
Why it matters here: This announcement expressly describes performance on METR’s tasks. It does not present the evaluation as covering every cognitive task.
- Publisher
- METR
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
What would settle it
An archived copy of the complete identified threads and original blog would permit a fuller assessment of the alleged unsupported generalizations.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, The latest AI scaling graph - and why it hardly makes sense
METR’s blog and its tweets (perhaps written by publicists or by generative AI rather than by scientists), however, dropped the careful qualifications and nuance of the technical paper and made claims that go far beyond anything that the study actually supports.
- Where
- paragraph 8
- Published
- 2025-05-03
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-the-latest-ai-scaling-graph-and-why-f680c2-a1: Current. Outcome: Mixed. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-the-latest-ai-scaling-graph-and-why-f680c2-a1.