“Let's make this really clear: AI cannot "just handle coding."”
Our reading:
AI cannot take over coding on its own.
Assessment: Supported
What happened
Was it true for the period described?
Outcome: Supported
The evidence matches what they said.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
The contract reads the claim as: as of the post date (August 25, 2025), AI could not reliably do software development on its own, so coding could not simply be handed to it. It does not read it as: AI cannot write any code. METR's independent evaluation of GPT-5 was published August 7, 2025. It estimated a 50% chance of success on software tasks that take a professional about 2 hours 17 minutes (95% CI 65 minutes to 4 hours 25 minutes) (e1, e2). It put the 80% time horizon at about 25 minutes (95% CI 8 to 65 minutes), compared with 20 minutes for OpenAI o3 and Claude Opus 4 (e2). These results show useful coding capability. They also show frequent failures on software tasks of modest length, which meets the contract's support condition. The packet contains no evidence from that period of AI reliably completing software development end to end without human coding expertise.
Caveats
- The claim states no threshold for reliability or task size, so the result depends on the plain reading of 'just handle coding'.
- The evidence comes from one evaluator's task suite. Its tasks differ from full product development.
- The search was not exhaustive for individual cases of non-coders building working software with AI.
- This assessment covers only the period just before the post (July 1 to August 25, 2025).
Evidence (2 items)
Supports the claim: METR,
we estimate it has a 50% chance of succeeding at software engineering tasks that would take human professionals (who are unfamiliar with the particular codebase) around 2 hours and 17 minutes.
Why it matters: The leading frontier model at the post date failed about half the time on software tasks a professional finishes in roughly two hours. That fits the claim that AI could not simply be handed coding work.
- Where
- Executive Summary
- Publisher
- METR
- Kind of source
- Independent measurement
- About events on
- Published
Supports the claim: METR,
We find the 80%-time horizon to be around 25m (8m - 65m 95% CI), compared to OpenAI o3’s and Claude Opus 4’s 20m.
Why it matters: Frontier models reached 80% reliability only on tasks of about 20 to 25 minutes of professional work. That is far from reliably handling software development on their own.
- Publisher
- METR
- Kind of source
- Independent measurement
- About events on
- Published
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Where we had to interpret
- Plain reading: as of the post date, AI systems could not on their own reliably do the work of software development, so that a person without coding skill could hand coding over to them and get working software. The claim is not that AI cannot write any code.
- He describes the current state without naming a period. I use the period just before the post (July 1 to August 25, 2025), when the latest frontier models had been released and independently measured.
- Independent measurements of how reliably frontier AI agents complete realistic software-engineering tasks on their own are fair evidence.
Right if: Independent evaluations from around the post date show that frontier AI agents fail often on realistic software tasks of modest length, so they cannot be relied on to handle coding on their own.
Wrong if: Credible evidence from around the post date shows AI systems reliably completing realistic software development work end to end, without human coding expertise, at a level where coding could be handed off to them.
We wrote this test after the outcome was known.
We found 37 claims in this post. See them all.
Full record
Source and dates
Where's Your Ed At, How To Argue With An AI Booster, paragraph 397.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-28. The post may have been edited after 2025-08-25.
Where it was said (1)
Said once in the posts we read.
: Where's Your Ed At, How To Argue With An AI Booster
Let's make this really clear: AI cannot "just handle coding."
- Where
- paragraph 397
- Published
- 2025-08-25
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Supported. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.