“There is plenty of headroom left on the current METR set of tasks if you simply demand 80% success, even more headroom if you demand 95% success.”
Our reading, the sentence that was checked:
There is plenty of headroom left on the current METR task set if you demand 80% success, and even more headroom if you demand 95% success.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Misplaced panic over AI progress, paragraph 18.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-misplaced-panic-over-ai-progress-5125cd-a1
We recorded 14 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2026-05-10. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- if you demand 80% success (qualifier, stated by the speaker)
- even more if you demand 95% success (qualifier, stated by the speaker)
From the speaker
- current METR set of tasks
- plenty of headroom
- 80% success
- even more headroom
- 95% success
Our interpretation
- Headroom means that the existing suite still contains substantially longer tasks than Mythos's estimated horizon at the specified reliability.
- The comparison concerns the early Mythos Preview version underlying the graph discussed in the post.
- The target period covers the latest underlying evaluation through publication, rather than subsequent Mythos versions.
- The 95% comparison can be supported as an implication of METR's fitted success curve; this does not establish a separately validated 95% horizon or count of tasks meeting that reliability.
Counts as supported if: The 80% horizon lies substantially below the existing suite's upper measurable task lengths, and the fitted success curve implies a shorter horizon at 95%.
Counts as contradicted if: The suite is already effectively saturated at 80%, or reliable results show no additional room when the success requirement rises to 95%.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The ordinary graph-based reading is supported. Mythos's fitted 80% horizon was about 3.1 hours, substantially below the suite's 16-hour reliable measurement limit. The same fitted curve implies a shorter horizon at 95%, leaving more room on the existing task-duration scale. This supports the comparative headroom claim, but does not establish how many individual tasks remain below either reliability threshold. The figures and method come from [METR's results](https://metr.org/time-horizons/).
Limitations
- Support depends on reading headroom as room on the fitted horizon scale.
- The calculated 95% value is illustrative and lacks a validated uncertainty interval.
- Results concern an early Mythos Preview checkpoint, not every subsequent version.
Evidence (2 items)
Supports the claim: METR,
logistic curve
Why it matters here: The linked official data give an 80% horizon of about 3.10 hours, well below the page's 16-hour upper reliable measurement range. Applying the documented fitted curve implies a still shorter 95% horizon; that implication is an evaluator calculation, not a published validated 95% measurement.
- Publisher
- METR
- Kind of source
- Independent measurement
- Arithmetic
- H80_hours = 185.911829 / 60 = 3.09853048. For the fitted logistic curve, H95_minutes = H50 * (H80/H50)^(ln(19)/ln(4)) = 26.70861024. (inputs: Official linked data: https://metr.org/assets/benchmark_results_1_1.yaml; claude_mythos_preview_early_inspect p50_horizon_length estimate: 1044.780145 minutes; claude_mythos_preview_early_inspect p80_horizon_length estimate: 185.911829 minutes; METR's analysis repository specifies logistic success probability against log2(human_minutes).; 80% horizon rounded to 3.10 hours; illustrative fitted 95% horizon rounded to 26.7 minutes.)
- Describes events of
- Public from
Context only: METR,
March 2026.
Why it matters here: Identifies the checkpoint and testing period. The announcement reports a 50% horizon of at least 16 hours and warns that this approaches the suite's measurement limit.
- Publisher
- METR
- Kind of source
- Independent measurement
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Misplaced panic over AI progress
There is plenty of headroom left on the current METR set of tasks if you simply demand 80% success, even more headroom if you demand 95% success.
- Where
- paragraph 18
- Published
- 2026-05-10
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-misplaced-panic-over-ai-progress-5125cd-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-misplaced-panic-over-ai-progress-5125cd-a1.