Form, function, and the giant gulf between drawing a picture and understanding the world
We recorded 6 claims from this post. 5 were checked against evidence. 1 can’t be checked. 5 more are held. Read the post at its source.
Claims from this post (6)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“In assessing progress towards general intelligence, the critical question should be, how much do systems like Dall-E, Imagen, Midjourney, and Stable Diffusion really understand the world, such that they can reason on and act on that knowledge?”
Our reading: Progress towards general intelligence from image systems like DALL-E, Imagen, Midjourney and Stable Diffusion should be assessed by how much they understand the world so they can reason and act on it.
No horizon
Qualifier: in assessing progress towards general intelligence
Assessment: UnresolvedSaid
Observation
“On #1, the answer is a clear yes; only highly trained human artists could do better.”
Our reading: Image synthesis systems generate high quality images that only highly trained human artists could surpass.
No horizon Evidence as of
Limitation: Image quality can vary by task and evaluator.
Assessment: SupportedSaid
Observation
“On #2, the answer is mixed. They do well on some inputs (like astronaut rides horse) but more poorly on others (like horse rides astronaut, which I discussed in an earlier post).”
Our reading: Image synthesis systems only inconsistently match their linguistic input to the images produced, doing well on some prompts and poorly on others.
No horizon Evidence as of
Limitation: The strongest experimental evidence covers DALL-E 2.
Assessment: UnresolvedSaid
Observation
“Nine more tries, and only one very marginal success (top right corner):”
Our reading: Stable Diffusion produced only one very marginal success in nine further attempts at a part-function prompt.
No horizon Evidence as of
Limitation: Success grading includes a subjective boundary.
Assessment: UnresolvedSaid
Observation
“Negation is, as ever, a problem.”
Our reading: Text-to-image systems like Stable Diffusion continue to have problems with negation in prompts.
No horizon Evidence as of
Limitation: This is an evidence gap, not a finding that negation worked reliably.
Assessment: UnresolvedSaid
Observation
“Even “draw a white bicycle with green wheels”, which focuses purely on part-whole relationships without function or complex syntax, is problematic:”
Our reading: Stable Diffusion has trouble even with simple part-whole prompts such as a white bicycle with green wheels.
No horizon Evidence as of
Limitation: Failure to locate independent records does not show that Marcus's displayed results were inaccurate.
Held from this post (5)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: From what Marcus can tell, Craiyon (formerly DALL-E Mini) is completely at sea on prompts that test understanding of parts, wholes and function.
- Attribution is uncertain