“It will often do what you want, sometimes not—and it will remain difficult to anticipate which in advance..”
Our reading:
It will remain difficult to anticipate in advance whether GPT-5 will do what the user wants.
Assessment: Unresolved
What happened
Did the forecast come true?
Outcome: Unresolved
We checked, and the evidence so far doesn’t settle it.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
On August 7, 2025, OpenAI reported 89% accuracy for GPT-5 on BrowseComp Long Context inputs of 128K–256K tokens [e1]. ORCFLO's May 10, 2026 benchmark reported an Output Consistency score of 96.0 for repeated runs [e2]. Neither result measures whether users can predict individual successes and failures before seeing GPT-5's answers [e1, e2]. These results therefore do not settle the forecast about difficulty anticipating whether GPT-5 will satisfy a request [e1, e2].
Caveats
- The accuracy result does not show whether users could identify likely failures in advance [e1].
- ORCFLO's reported consistency score does not establish users' ability to anticipate whether an answer will satisfy their request [e2].
Evidence (2 items)
Context only: OpenAI,
GPT‑5 gives the correct answer
Why it matters: The reported benchmark contains both successes and failures. It does not test whether users could identify those failures beforehand, so it cannot settle the central forecast.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- About events on
- Published
Context only: ORCFLO,
predictable behavior across repeated runs
Why it matters: This benchmark reports strong repeatability, which weighs against an unrestricted claim of unpredictable behavior. Its consistency score is not a measured rate of users correctly anticipating success, and its 40 English text-only cases exclude several major task types.
- Publisher
- ORCFLO
- Kind of source
- Benchmark result
- About events on
- Published
How we judged it
- Kind of claim
- Forecast
- Deadline
- None given. We don’t add one.
Where we had to interpret
- The plain reading is that users will have difficulty predicting, before seeing an answer, whether GPT-5 will satisfy their request.
- The claim concerns the GPT-5 release discussed in the post. Results for later GPT-5.x models do not automatically settle it.
- Mixed success rates establish that failures occur, but do not by themselves establish that those failures are difficult to anticipate.
- Marcus supplies neither a numerical threshold for difficulty nor a task distribution. Representative evaluations of advance predictions would be more informative than isolated failures.
Right if: Evaluations of released GPT-5 show that users or practical advance indicators have substantial difficulty distinguishing requests it will satisfy from requests it will fail, across a reasonably representative range of tasks.
Wrong if: Evaluations show that users can reliably distinguish likely successes from failures in advance across a reasonably representative range of requests, with uncertainty communicated accurately enough to guide use.
We wrote this test after the outcome was known.
What would settle it
A reproducible evaluation comparing advance predictions by users or practical indicators with original GPT-5's actual outcomes across representative requests. It should specify model settings and success criteria, use repeated trials, and report uncertainty.
We found 18 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, What to Expect When You’re Expecting … GPT-5, paragraph 3.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-28. The post may have been edited after 2025-07-27.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, What to Expect When You’re Expecting … GPT-5
It will often do what you want, sometimes not—and it will remain difficult to anticipate which in advance..
- Where
- paragraph 3
- Published
- 2025-07-27
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Unresolved. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
The final check failed: a cited source didn’t say what the write-up said, and the write-up said more than the evidence showed.