“Chance is definitely part of what’s going on. But it’s not quite the right way to understand the juxtaposition of brilliance and stupidity that we see within GPT.”
Our reading, the sentence that was checked:
Chance plays a part, but it does not properly explain why GPT mixes brilliance and stupidity.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, How come GPT can seem so brilliant one minute and so breathtakingly dumb the next?, paragraph 23.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-how-come-gpt-can-seem-so-brilliant-b0d5e4-a1
We recorded 9 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2022-12-01. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Chance is definitely part
- not quite the right way to understand
- juxtaposition of brilliance and stupidity
- within GPT
Our interpretation
- Read in context, the claim rejects chance alone as an explanation, rather than denying that generation uses randomness.
- The ordinary reading is that learned structure produces systematic linguistic competence without guaranteeing correct answers.
- The relevant systems are GPT-3 and its contemporary derivatives, not later GPT models.
Counts as supported if: Documented generation methods and controlled comparisons show that learned structure systematically affects fluent output and errors, while randomness also contributes to generation.
Counts as contradicted if: Controlled evidence shows that chance alone accounts for the apparent competence and failures, with learned structure making no material contribution.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
GPT's output depended systematically on learned structure, conditioning, and training. Sampling introduced randomness, but controlled generation experiments and improvements following additional training show why chance alone is inadequate. This supports Marcus's limited claim without establishing his subsequent explanation of every error.
Limitations
- The degraded control changed model size, context, and randomness together.
- The evidence does not quantify how much of any individual response is attributable to sampling.
- The training results were reported by the developer.
Evidence (2 items)
Supports the claim: Brown et al., arXiv,
a 160M parameter model with no context and increased output randomness.
Why it matters here: Human detection accuracy was about 86% for the degraded control and 52% for GPT-3, with outputs selected programmatically. This supports systematic competence beyond random text, although the control changes several factors together.
- Publisher
- Brown et al., arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: OpenAI,
They also make up facts less often
Why it matters here: The disclosure describes next-word training and subsequent training with human feedback, which improved instruction following and reduced fabricated facts. These systematic effects support learned mechanisms as an explanation beyond chance alone.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, How come GPT can seem so brilliant one minute and so breathtakingly dumb the next?
Chance is definitely part of what’s going on. But it’s not quite the right way to understand the juxtaposition of brilliance and stupidity that we see within GPT.
- Where
- paragraph 23
- Published
- 2022-12-01
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-how-come-gpt-can-seem-so-brilliant-b0d5e4-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-how-come-gpt-can-seem-so-brilliant-b0d5e4-a1.