Image generation: Still crazy after all these years
We recorded 4 claims from this post. 3 were checked against evidence. 1 can’t be checked. 6 more are held. Read the post at its source.
Claims from this post (4)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Causal claim
“The point was that understanding unusual phrases requires more than simple statistical matching.”
Our reading: Understanding unusual phrases requires more than simple statistical matching.
No horizon
Qualifier: unusual phrases
Assessment: UnresolvedSaid
Observation
“Not much better on some other tests of noncanonical images (which push language understanding harder), such as this one, adapted from an example Sandeep Manudhane sent me:”
Our reading: Imagen 4 is not much better on some other tests of noncanonical images, which push language understanding harder.
No horizon Evidence as of
Limitation: The comparison and degree of improvement are imprecise.
Assessment: UnresolvedSaid
Observation
“Text in general is better now (though not in this image) but watch comprehension of any time other than 10:10 (most frequent in advertisements) remains dreadful.”
Our reading: Imagen 4 is better at text in general now, but its handling of watch times other than 10:10 remains dreadful.
No horizon Evidence as of
Limitation: The packet contains no representative watch-time evaluation matched to the June 2025 Imagen 4 version.
Assessment: UnresolvedSaid
Observation
“Now? Here’s what I got out of Imagen 4, showing failure at the feature “binding problem” that some of us have been harping on for three decades:”
Our reading: Imagen 4 still fails at the feature binding problem on the stacked colored blocks prompt.
No horizon Evidence as of
Limitation: The decisive artifact is an image that the retrieved plain text does not make inspectable.
Held from this post (6)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews reached different assessments. Our reading, not assessed: Imagen 4 is far better than the original in some ways, especially text rendering, but failed all four attempts to draw a horse riding an astronaut.
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: o3 does better on some old examples but still cannot correctly generate the new woman-and-watch image.
- The criteria are ambiguous. Our reading, not assessed: Even three years later, you can't count on Imagen to correctly interpret any given prompt, because its language understanding is still far from perfect.
- The criteria are ambiguous. Our reading, not assessed: Like Sora, Google's Veo failed to generate correct video of a person handwriting a phrase in a planner.
- The criteria are ambiguous. Our reading, not assessed: ChatGPT could not generate a tree on top of a monkey, even after acknowledging its own error.