“On #2, the answer is mixed. They do well on some inputs (like astronaut rides horse) but more poorly on others (like horse rides astronaut, which I discussed in an earlier post).”
Our reading, the sentence that was checked:
Image synthesis systems only inconsistently match their linguistic input to the images produced, doing well on some prompts and poorly on others.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Form, function, and the giant gulf between drawing a picture and understanding the world, paragraph 8.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-form-function-and-the-giant-gulf-52ece7-a1
We recorded 6 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2022-09-18. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- the answer is mixed
- do well on some inputs
- more poorly on others
- astronaut rides horse
- horse rides astronaut
Our interpretation
- The plain reading is uneven correspondence between prompts and generated images, including difficulty with relationships and role reversals.
- The horse examples illustrate the general claim; they do not assert that every generation of either prompt must have the same result.
- The surrounding discussion concerns the generation of systems available since DALL-E 2 appeared in 2022, through the post date. April through September 18 is the assessment window.
- Support requires documented successes as well as failures, rather than attractive images alone.
Counts as supported if: Contemporaneous evaluations document successful prompt matching alongside substantial failures on other prompts, especially compositional or relational instructions.
Counts as contradicted if: Representative contemporaneous evaluations show consistently accurate correspondence across ordinary and relational prompts, with the alleged failure pattern absent.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Contemporaneous independent research found successful prompt matches alongside frequent relational failures. MIT's September report also describes attribute-binding errors in DALL-E 2. These findings support Marcus's general observation that correspondence with linguistic input was uneven. They do not establish identical behavior across all systems or a failure rate for the exact horse-and-astronaut examples.
Limitations
- The strongest experimental evidence covers DALL-E 2.
- The relational study was a preprint.
- These output tests do not by themselves establish whether a system understands the world.
Evidence (2 items)
Supports the claim: arXiv; Colin Conwell and Tomer Ullman,
Overall, we find that only ~22% of images matched basic relation prompts.
Why it matters here: The study's human judgments show some successful matching alongside widespread relational failures. Its results were reported by July; the precise image-generation dates are not supplied.
- Where
- Abstract
- Publisher
- arXiv; Colin Conwell and Tomer Ullman
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: MIT News / MIT CSAIL,
red truck and green house
Why it matters here: MIT's report describes DALL-E 2 swapping requested colors between objects while also reporting its strength at natural-image generation. This independently corroborates uneven correspondence with linguistic instructions.
- Publisher
- MIT News / MIT CSAIL
- Kind of source
- News report
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Form, function, and the giant gulf between drawing a picture and understanding the world
On #2, the answer is mixed. They do well on some inputs (like astronaut rides horse) but more poorly on others (like horse rides astronaut, which I discussed in an earlier post).
- Where
- paragraph 8
- Published
- 2022-09-18
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-form-function-and-the-giant-gulf-52ece7-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-form-function-and-the-giant-gulf-52ece7-a1.