“They (often) don’t come up with a novel independent design that would meet the requirements.”
Our reading, the sentence that was checked:
GenAI systems often don't come up with a novel independent design that would meet the requirements.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Further Trouble in Hinton City, paragraph 9 (item 2 of 3).
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-further-trouble-in-hinton-city-ebe684-a2
We recorded 4 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-02-08. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- often (qualifier, stated by the speaker)
From the speaker
- They (often) don’t
- a novel independent design
- meet the requirements
- design to used with only one hand
Our interpretation
- In context, they refers principally to contemporary generative image systems given unfamiliar design briefs.
- Often means a recurring tendency, not an invented requirement that more than half of all outputs fail.
- Success requires both meaningful novelty and compliance with the brief's functional constraints.
- The target is the latest publicly documented performance of systems available before publication; no fixed reporting interval or starting date is supplied.
Counts as supported if: Repeated evaluations across relevant design briefs show substantial recurring failures to produce both novel and functionally compliant designs.
Counts as contradicted if: Representative evaluations show reliable novel, compliant designs and establish that the alleged failures are isolated or unrepresentative.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The HRS-Bench version available November 23, 2023 reports difficulties generating requested object counts, visual text and grounded emotions [e1]. NLeseul's demonstration, published February 4, 2023, reports failures and partial successes on unusual object prompts [e2]. Neither item establishes how often systems available before February 8, 2024 failed to produce designs that were both novel and functionally compliant [e1, e2].
Limitations
- The packet lacks representative evaluations jointly measuring novelty and functional compliance.
- The controller demonstration's complete sampling record and denominator were not verified.
- Selected failures and partial successes cannot establish the claimed recurring tendency.
- Visual prompt compliance alone does not establish functional usability.
Evidence (2 items)
Context only: arXiv; Bakr and colleagues,
often struggle
Why it matters here: The benchmark documents relevant instruction-following failures and separately studies creative generation. These measures do not directly establish how often novel functional designs satisfy product briefs.
- Publisher
- arXiv; Bakr and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: NLeseul, Thinking About Stuff,
better
Why it matters here: The original demonstrator reports partial successes as well as failures on unusual object prompts. This cautions against inferring inability from selected failures, but the demonstration does not provide a representative frequency estimate.
- Publisher
- NLeseul, Thinking About Stuff
- Kind of source
- Demonstration
- Describes events of
- Public from
What would settle it
Representative evaluations of systems available before February 8, 2024 across unfamiliar design briefs, recording model versions, prompts, permitted prompting effort and every sampled output, with separate assessments of novelty and functional compliance.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Further Trouble in Hinton City
They (often) don’t come up with a novel independent design that would meet the requirements.
- Published
- 2024-02-08
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-further-trouble-in-hinton-city-ebe684-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-further-trouble-in-hinton-city-ebe684-a2.