“In the particular case, we know that the system succeeds at previously-discussed prompts but does less well on ones not previously-discussed. That tells us something about the internal mechanism, and suggests that it is far from robust.”
Our reading:
GPT-4o succeeds on previously discussed prompts but does worse on new ones, suggesting its compositional ability is far from robust.
Assessment: Unresolved
What happened
Was it true for the period described?
Outcome: Unresolved
We checked, and the evidence so far doesn’t settle it.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
The publicized-prompt successes are documented, and independent research confirms remaining compositional failures. But neither fact establishes that comparable fresh prompts performed worse. Alexander's harder example changes complexity as well as familiarity. Other research also reports substantial compositional strengths. The central comparison therefore remains unverified; it is not contradicted merely because evidence is missing. The records justify caution about robustness but do not settle Marcus's full claim.
Caveats
- An anecdotal reading has some support, but the ordinary comparative reading requires evidence about comparable prompts.
- The bet account is a primary demonstration, not a controlled comparative study.
- No arbitrary July-only testing window was used.
- Publication of component findings does not make the full comparative outcome known.
Evidence (3 items)
Supports the claim: Scott Alexander, Astral Codex Ten,
The bet is now over, and official judge Gwern agrees I’ve won.
Why it matters: Alexander reports that the five publicized prompts all succeeded on their first attempts in images generated June 1, supporting the familiar-prompt component. His separate failure example combines more requirements, so it does not establish worse performance on comparable fresh prompts.
- Publisher
- Scott Alexander, Astral Codex Ten
- Kind of source
- Demonstration
- About events on
- Published
Context only: Zhiyuan Yan and colleagues, arXiv,
These limitations reflect the model’s difficulty in spatial reasoning and maintaining image consistency under high visual complexity.
Why it matters: The researchers document malformed poses and implausible overlaps, supporting concern about remaining compositional limitations. They do not compare the bet prompts with comparable previously undiscussed prompts, so these failures cannot establish the asserted gap.
- Publisher
- Zhiyuan Yan and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
Context only: An Empirical Study of GPT-4o Image Generation Capabilities, arXiv,
GPT-4o displays impressive compositional abilities, accurately assembling complex scene elements, styles, or attributes described in prompts.
Why it matters: This independent study reports substantial compositional strengths beyond the specific bet demonstration, while also discussing failures. It qualifies the broad robustness inference but does not measure the claimed publicized-versus-fresh performance gap.
- Publisher
- An Empirical Study of GPT-4o Image Generation Capabilities, arXiv
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Where we had to interpret
- The system is GPT-4o image generation, as identified in the surrounding discussion.
- The plain reading is that success on the publicized prompts does not transfer reliably to comparable fresh compositional prompts. Marcus supplies no numerical performance threshold.
- The target is the system's demonstrated performance as of July 9, 2025. The start is unspecified: relevant earlier testing of this system is eligible, including the testing behind Alexander's retrospective. No July-only window is imposed.
- Comparable means broadly similar compositional demands and generation and grading procedures. It does not require an identical benchmark or proof that a prompt never appeared anywhere online.
- The statement about internal mechanism is a tentative inference from the asserted performance gap, not a definite claim that training contamination occurred.
Right if: Independent records show success on the previously discussed prompts and a meaningful, recurring performance drop on comparable fresh compositional prompts using the same image-generation system. The contrast should demonstrate unreliable transfer beyond the publicized examples.
Wrong if: Adequate comparative testing shows that the same system performs similarly or better on comparable fresh prompts, with reliable transfer of the relevant compositional abilities.
We wrote this test after the outcome was known.
What would settle it
Independent records from the relevant GPT-4o image-generation system comparing the publicized prompts with fresh prompts of similar compositional difficulty, with outputs, attempt counts, and consistent grading. A recurring gap would support the claim; reliable comparable performance would contradict it.
We found 9 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, Scott Alexander’s Misleading Victory Lap, paragraph 26.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-28. The post may have been edited after 2025-07-09.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, Scott Alexander’s Misleading Victory Lap
In the particular case, we know that the system succeeds at previously-discussed prompts but does less well on ones not previously-discussed. That tells us something about the internal mechanism, and suggests that it is far from robust.
- Where
- paragraph 26
- Published
- 2025-07-09
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Unresolved. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
A reviewer said our test wasn’t fair to what was actually said.