“What this means for business and society is that you can’t simply drop o3 or Claude into some complex problem and expect it to work reliably.”
Our reading, the sentence that was checked:
o3 or Claude cannot simply be dropped into complex problems and expected to work reliably.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, A knockout blow for LLMs?, paragraph 34.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-a-knockout-blow-for-llms-27f761-a3
We recorded 19 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-06-07. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- dropped into complex problems (qualifier, stated by the speaker)
From the speaker
- can’t simply drop o3 or Claude into some complex problem
- expect it to work reliably
Our interpretation
- The ordinary reading warns against expecting dependable completion of arbitrary complex problems without task-specific engineering or oversight. It does not say every complex problem fails.
- Basic interfaces that let a model act in an environment are relevant to simple deployment, but their results are not identical to unassisted chat.
- Assess o3 and contemporaneous Claude versions separately; o3-mini is not a substitute for evidence about o3.
- Use calendar 2025 through June 7 for contemporary capability reports because no period or regular reporting interval is specified.
- Dropped into complex problems describes the deployment setting, not an antecedent for a conditional forecast.
Counts as supported if: Evaluations of o3 and contemporaneous Claude versions show substantial failure rates on verifiable complex tasks without bespoke task solutions or continuing human correction.
Counts as contradicted if: Broad evaluations demonstrate dependable completion of representative complex tasks by both named model families under simple deployment conditions.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
METR's April 2025 evaluation places a pre-release o3 checkpoint at 50% predicted success on HCAST tasks taking humans about 1.5 hours [e1]. Its March 2025 study reports Claude 3.7 Sonnet's 50% success horizon at about 59 minutes and its 80% horizon at about 15 minutes of human task time [e2]. These results support the warning against expecting dependable complex-task completion in the evaluated deployment settings [e1, e2].
Limitations
- The evaluations use agent interfaces and do not directly test exclusively unassisted chat.
- The o3 result concerns a pre-release checkpoint; the Claude result concerns 3.7 Sonnet.
- The task suites differ, so the reported horizons are not directly comparable [e1].
- The evidence does not cover every business problem or establish a capability ceiling.
Evidence (2 items)
Supports the claim: METR,
50% time-horizon score
Why it matters here: The evaluated o3 checkpoint reached only 50% predicted success at tasks taking humans about 1.5 hours, despite setting a new high in this evaluation. This supports caution about dependable complex-task completion, while the pre-release checkpoint and agent interface limit direct application to unassisted released o3.
- Where
- Summary of Results
- Publisher
- METR
- Kind of source
- Independent measurement
- Describes events of
- Public from
Supports the claim: METR, arXiv,
cannot reliably perform
Why it matters here: The section reports Claude 3.7 Sonnet's approximately 59-minute horizon at 50% success and approximately 15-minute horizon at 80% success. These substantial failure rates support the reliability warning for that Claude version.
- Publisher
- METR, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, A knockout blow for LLMs?
What this means for business and society is that you can’t simply drop o3 or Claude into some complex problem and expect it to work reliably.
- Where
- paragraph 34
- Published
- 2025-06-07
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-a-knockout-blow-for-llms-27f761-a3: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.