“And this model was clearly better than the one I had ridiculed.”
Our reading:
The model that produced the rebuttal image (ChatGPT-5 Thinking) was clearly better than the model whose unicycle image Marcus had ridiculed.
Assessment: Unresolved
What happened
Was it true for the period described?
Outcome: Unresolved
We checked, and the evidence so far doesn’t settle it.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
Marcus made a comparison of models. The record located does not identify the original model or establish a consistent performance difference between the two models. Even accepting his description of the rebuttal image as sharper and more accurate in some details, that would establish only a difference between selected outputs. The available evidence therefore neither supports nor contradicts the model-level claim.
Caveats
- Search coverage is incomplete, and the comments page could not be read with the required tool.
- The missing comparison is an evidence gap, not evidence that Marcus was wrong.
- No numerical meaning of clearly better was supplied.
Evidence (1 item)
Context only: Gary Marcus, Marcus on AI,
And at first I was stunned by the refutation, produced by ChatGPT-5 Thinking.
Why it matters: This identifies Marcus's stated attribution of the rebuttal image. It neither independently verifies that attribution nor establishes superiority over the original model.
- Publisher
- Gary Marcus, Marcus on AI
- Kind of source
- Secondary summary
- About events on
- Published
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Where we had to interpret
- The claim compares the capabilities of two models. A comparison of two selected images alone does not establish it.
- In context, the ordinary reading of better is better at generating visually clear images with accurately depicted objects, including the unicycle. It does not require superiority at every task or a better understanding of the world.
- Clearly better means a meaningful, consistent advantage under comparable conditions. Marcus supplies no numerical threshold.
- The target date is August 31, 2025, because the post describes the image exchange as occurring that morning. The comparison concerns the model versions used then.
Right if: Identify both model versions and establish a clear, consistent advantage for the rebuttal model in relevant image-generation quality through a credible comparison covering multiple outputs under comparable conditions.
Wrong if: Identify both model versions and establish through a sufficiently informative comparison that the rebuttal model was equivalent or worse in relevant image-generation quality. Mere failure to find evidence of superiority does not contradict the claim.
We wrote this test after the outcome was known.
What would settle it
Verified identities and configurations for both models, plus a credible comparison of those historical versions across relevant image-generation tasks under comparable conditions, available by the evaluation date.
We found 5 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, Spinal Tap meets John Malkovich, paragraph 15.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-28. The post may have been edited after 2025-08-31.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, Spinal Tap meets John Malkovich
And this model was clearly better than the one I had ridiculed.
- Where
- paragraph 15
- Published
- 2025-08-31
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Unresolved. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
A reviewer said our test wasn’t fair to what was actually said.