“People building things like virtual assistants and agents will find that they cannot reliably enough map user language onto user intentions.”
Our reading:
Builders of virtual assistants and agents using GPT-5 will be unable to map user language onto user intentions reliably enough.
Assessment: Unresolved
What happened
Did the forecast come true?
Outcome: Unresolved
We checked, and the evidence so far doesn’t settle it.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
On August 7, 2025, OpenAI reported about 97% success for GPT-5 on τ2-bench telecom [e1]. METR reported that GPT-5's attempts on a small set of product tasks had more bugs and were less feature complete than human contractors' attempts [e2]. The contractors had 10–20 hours [e2]. CLARITY, published April 24, 2026, reports difficulties with ambiguous database queries and improvements from examples and structured metadata [e3]. These evaluations do not establish whether intent-mapping errors prevent GPT-5 assistants from meeting practical requirements across representative applications [e1, e2, e3].
Caveats
- OpenAI's aggregate benchmark score does not separately measure intent mapping [e1].
- METR's comparison does not establish which bugs or missing features resulted from misunderstood intent [e2].
- CLARITY's generated database tasks do not establish reliability against confirmed user intentions across assistant applications [e3].
Evidence (3 items)
Context only: OpenAI,
GPT‑5 scores 97%.
Why it matters: OpenAI reports high success on a tool-agent benchmark involving user interactions and changing environment state. This is meaningful evidence of practical capability, but the aggregate score does not separately measure intent mapping.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- About events on
- Published
Context only: METR,
GPT-5 attempts at these tasks contained more bugs and were less feature complete.
Why it matters: METR found weaknesses on a small set of tasks requiring inference of intended product behavior. Bugs and missing features can arise after correct understanding, so the finding does not isolate the claimed failure.
- Where
- Capability discussion, paragraph on manually scored tasks requiring inference of product intentions
- Publisher
- METR
- Kind of source
- Independent measurement
- About events on
- Published
Context only: Tabinda Sarwar and colleagues, arXiv,
LLMs struggle to enumerate multiple SQL interpretations for column-level ambiguity
Why it matters: The study includes GPT-5 and finds ambiguity-resolution limitations, with improvements from examples and structured metadata. Its generated database tasks do not establish a general deployment threshold or measure user-confirmed intentions across assistant applications; the date marks publication of the finding.
- Publisher
- Tabinda Sarwar and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
How we judged it
- Kind of claim
- Forecast
- Deadline
- None given. We don’t add one.
Where we had to interpret
- The surrounding paragraph makes original-GPT-5-based assistants and agents the target.
- The plain reading concerns correctly identifying a user's intended task, constraints, and desired actions well enough for practical deployment.
- Intent interpretation must be distinguished from execution, planning, coding, or tool failures after the intent has been correctly understood.
- Clarifying questions are permitted; the claim does not require guessing intentions that the user has not communicated.
- Reliably enough means meeting practical task requirements, not perfect performance on every request.
Right if: Representative GPT-5 assistant evaluations isolate intent-mapping errors against user-confirmed or independently validated intentions and show that these errors prevent meeting practical application requirements.
Wrong if: Representative GPT-5 assistants meet practical application requirements in evaluations against independently established user intentions, including ambiguous requests and clarification.
We wrote this test after the outcome was known.
What would settle it
A representative evaluation of original-GPT-5 assistants against independently established user intentions. It should allow clarification, separate interpretation errors from execution errors, and test whether assistants meet explicit practical acceptance criteria.
We found 18 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, What to Expect When You’re Expecting … GPT-5, paragraph 6.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-28. The post may have been edited after 2025-07-27.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, What to Expect When You’re Expecting … GPT-5
People building things like virtual assistants and agents will find that they cannot reliably enough map user language onto user intentions.
- Where
- paragraph 6
- Published
- 2025-07-27
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Unresolved. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
The final check failed: a cited source didn’t say what the write-up said, and the write-up said more than the evidence showed.