“Not much better on some other tests of noncanonical images (which push language understanding harder), such as this one, adapted from an example Sandeep Manudhane sent me:”
Our reading, the sentence that was checked:
Imagen 4 is not much better on some other tests of noncanonical images, which push language understanding harder.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Image generation: Still crazy after all these years, paragraph 22.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-image-generation-still-crazy-after-91f5cf-a1
We recorded 4 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-06-25. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- some other tests of noncanonical images (scope, stated by the speaker)
- tests that push language understanding harder (qualifier, stated by the speaker)
From the speaker
- Not much better
- some other tests of noncanonical images
- which push language understanding harder
- such as this one
Our interpretation
- The immediate comparison is with the unsuccessful horse-riding-astronaut trials, although the broader passage also compares generations of Imagen.
- The ordinary reading is that Imagen 4 also performed poorly on the specified unusual combinations of requirements.
- For the woman-and-watch prompt, success requires left-handed writing, a watch on the woman's right wrist, a readable 3:20 dial, and the requested inscription.
- Language understanding is assessed through adherence to the prompt, without inferring an internal cognitive mechanism.
- The target period is the June 24–25 testing episode described immediately before this claim.
Counts as supported if: Independently documented outputs from the described testing episode show substantial failures on the cited noncanonical prompts, including the woman-and-watch requirements.
Counts as contradicted if: Independent examination establishes that the cited outputs meet their important requirements and therefore do not display the reported poor performance. General benchmark gains or isolated successes on other prompts are insufficient.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Google acknowledges difficulties in relevant categories, but that does not verify these particular outputs or their comparative failure rate. The searches did not locate independent evidence for the specified woman-and-watch test. The claim is testable in its context, but the available records do not settle it.
Limitations
- The comparison and degree of improvement are imprecise.
- General composition benchmarks cannot establish what the cited images showed.
Evidence (1 item)
Context only: Google DeepMind,
spatial reasoning and complex language
Why it matters here: Google identifies spatial reasoning and complex language among difficult prompt categories. This earlier general disclosure does not establish comparable failure on Marcus's particular tests.
- Publisher
- Google DeepMind
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
What would settle it
Independent scoring of authenticated June 2025 outputs for the cited prompts, with prompts, model version, trial counts, and the comparison set preserved.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Image generation: Still crazy after all these years
Not much better on some other tests of noncanonical images (which push language understanding harder), such as this one, adapted from an example Sandeep Manudhane sent me:
- Where
- paragraph 22
- Published
- 2025-06-25
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-image-generation-still-crazy-after-91f5cf-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-image-generation-still-crazy-after-91f5cf-a1.