“The question is not whether a clever user with a lot of experience and time to kill can induce a mission to do something correctly, but whether the systems reliably deliver what they are asked for. They don’t.”
Our reading, the sentence that was checked:
The large language model systems discussed in the post do not reliably deliver what they are asked for, even though clever, experienced users with time to spare can sometimes induce them to do something correctly.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, There Must be Some Misunderstanding, paragraph 18 (footnote 1).
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-there-must-be-some-misunderstanding-d102da-a2
We recorded 8 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-02-08. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- the systems discussed in the post (large language models) (scope, stated by the speaker)
- clever prompters with elaborate prompts can no doubt sometimes get correct results (qualifier, stated by the speaker)
From the speaker
- whether the systems reliably deliver what they are asked for
- They don’t.
- a clever user with a lot of experience and time to kill
- Compositionality remains a problem. So does negation.
Our interpretation
- The plain reading is that the contemporary systems repeatedly fail straightforward requests without extensive prompting or retries. It is not that every request fails.
- The surrounding examples concern image generation, especially composition and negation. Failures in an image-generation pipeline do not by themselves establish failures in every language-model task.
- No numerical reliability threshold or representative task distribution is specified.
- I use February 1–8, 2024, the publication month through the post date, as the latest observable period for this ongoing-state claim. The references to examples from last night and the author's current attempt anchor it to contemporary use.
- The concession about clever prompting is not a conditional forecast or an antecedent that must occur before the observation can be assessed.
Counts as supported if: Independent repeated tests of the systems operating in the target period show recurring failures to fulfill straightforward composition, negation, or comparable instructions without elaborate prompting. Isolated selected failures are insufficient.
Counts as contradicted if: Independent representative repeated tests of the same systems and relevant instructions show consistently correct fulfillment with ordinary prompts, and establish that the alleged recurring failures do not replicate.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
November 2023 GPT-4 text tests recorded both instruction-following successes and failures [e1]. A DALL-E 3 study conducted from March–November 2024 found recurring failures on negation prompts and improvement with prompt modification [e2]. Neither study measures the February 1–8, 2024 target period, so these results do not settle whether the systems discussed reliably fulfilled requests during that period [e1, e2].
Limitations
- The packet contains no qualifying independent repeated tests from the target period.
- The exact system versions, prompt rewriting, and complete interactions behind the post's examples are unverified.
- Image-generation results cannot establish reliability across language-model tasks.
- The claim specifies neither a numerical reliability threshold nor a representative distribution of requests.
Evidence (2 items)
Context only: arXiv; Jeffrey Zhou and colleagues,
through API calls.
Why it matters here: IFEval reports GPT-4 prompt-level accuracy of 76.89% under strict scoring and 79.30% under loose scoring, showing both substantial success and recurring failures. These are November 2023 text-generation tests, not measurements of the February 2024 image-generation pipeline.
- Publisher
- arXiv; Jeffrey Zhou and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Colin Conwell, Rupert Tawiah-Quashie, and Tomer Ullman,
March to November 2024.
Why it matters here: The later study reports mean image–prompt agreement of 12.3% for unmodified negation prompts and 40.7% for modified prompts. This corroborates a related failure pattern and shows improvement from prompt modification, but its stated testing dates exclude the target period; the year-level event date summarizes that multi-month testing period.
- Publisher
- arXiv; Colin Conwell, Rupert Tawiah-Quashie, and Tomer Ullman
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Independent repeated tests from February 1–8, 2024 of the systems discussed, documenting versions, interfaces, prompts, complete outputs, attempt counts, and scoring rules. Tests should include successes and failures on straightforward composition, negation, or comparable instructions and distinguish ordinary prompts from automatic rewriting and elaborate manual prompting.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, There Must be Some Misunderstanding
The question is not whether a clever user with a lot of experience and time to kill can induce a mission to do something correctly, but whether the systems reliably deliver what they are asked for. They don’t.
- Published
- 2024-02-08
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-there-must-be-some-misunderstanding-d102da-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-there-must-be-some-misunderstanding-d102da-a2.