“It will still make a significant number of shake-your-head stupid errors, in ways that are hard to fully predict.”
Our reading, the sentence that was checked:
GPT-4 will make a significant number of hard-to-predict stupid errors.
Assessment: Supported Settled before its deadline on 2026-10-29 Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, What to Expect When You’re Expecting … GPT-4, prediction 1 of 7.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-what-to-expect-when-youre-expecting-6a17e4-a1
We recorded 27 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2022-12-25. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Forecast
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- Settled before its deadline
- Deadline state on 2026-10-29
- Settled before its deadline (the release’s reference date)
From the speaker
- a significant number
- shake-your-head stupid errors
- hard to fully predict
Our interpretation
- The target is GPT-4 itself, including its early released versions, rather than all subsequent OpenAI models.
- The plain reading is recurring elementary mistakes with unexpectedly uneven occurrence across similar tasks. It does not require errors to be mathematically unpredictable.
- Significant means a substantial recurring pattern in controlled evaluations, rather than isolated anecdotes. Marcus supplied no population-wide error-rate threshold.
Counts as supported if: Controlled GPT-4 evaluations find substantial elementary errors and unexpectedly large differences between closely related tasks or inputs.
Counts as contradicted if: Broad evaluations find elementary mistakes rare and show that their occurrence can reliably be anticipated across reasonable task variations.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Did the forecast come true?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- Settled before its deadline
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
GPT-4 made substantial numbers of objectively wrong answers on elementary operations, with large differences between closely related tasks. That supports the ordinary reading of Marcus's forecast. The predictability component is less direct: the study explains some broad failure patterns, so it does not establish that failures are inherently unpredictable.
Limitations
- The judgment depends on reading hard to predict as unexpectedly uneven performance, rather than requiring a direct prediction experiment.
- Results concern the tested GPT-4 version and prompting methods.
- Benchmark error rates are not population-wide usage rates.
Evidence (1 item)
Supports the claim: arXiv,
surprising failure modes
Why it matters here: GPT-4 achieved 76% accuracy joining first letters but 3% joining second letters, and 80% sorting words alphabetically but 32% in reverse order. These controlled differences support recurring elementary mistakes and unexpectedly uneven performance, although the authors also successfully predicted broad patterns from task frequency.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, What to Expect When You’re Expecting … GPT-4
It will still make a significant number of shake-your-head stupid errors, in ways that are hard to fully predict.
- Where
- prediction 1 of 7
- Published
- 2022-12-25
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-what-to-expect-when-youre-expecting-6a17e4-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-what-to-expect-when-youre-expecting-6a17e4-a1.