“But many people have claimed over the years to solve the problem, and none of those systems have proven to be reliable; every proposed solution has been like Clever Hans, working in dim light, leveraging large databases to some degree, but falling apart upon careful inspection; they might get 60% on some task, but they never really master it.”
Our reading, the sentence that was checked:
Every claimed solution to compositionality so far has proven unreliable and falls apart on careful inspection.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?, paragraph 32.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-did-googleai-just-snooker-one-of-8f381c-a1
We recorded 12 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2022-09-14. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- over the years
- none of those systems have proven to be reliable
- every proposed solution
- falling apart upon careful inspection
- they never really master it
Our interpretation
- The context concerns learned systems proposed as solutions to compositional understanding, through September 14, 2022.
- The ordinary reading of reliability is successful generalization across unfamiliar combinations under controlled testing. Marcus supplies no numerical threshold.
- Broad compositional understanding and mastery of a restricted benchmark are different commitments. Success on a restricted benchmark challenges the wording but does not establish a general solution.
- The historical interval has no stated starting date.
Counts as supported if: A sufficiently comprehensive record of proposed solutions shows that each fails controlled tests of the compositional ability it claims to solve.
Counts as contradicted if: At least one proposed solution publicly demonstrates reliable compositional generalization under careful testing, at the breadth implied by the surrounding discussion.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Controlled evaluations substantiate failures in some systems. But contemporary COGS results also demonstrate strong structural generalization within a restricted task. Neither establishes the status of every proposed broad solution. Calling the claim supported would overextend selected failures; calling it contradicted would require treating restricted benchmark success as the general solution discussed in context.
Limitations
- The universal scope exceeds the coverage of the retrieved evaluations.
- The proposed outcome depends materially on distinguishing broad understanding from restricted task mastery.
Evidence (2 items)
Supports the claim: arXiv; Brenden Lake and Marco Baroni,
RNNs fail spectacularly.
Why it matters here: These controlled experiments support the criticism for the tested recurrent networks. They do not cover every proposed compositionality solution.
- Where
- Abstract
- Publisher
- arXiv; Brenden Lake and Marco Baroni
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Pia Weißenhorn and colleagues,
maintain their high accuracy
Why it matters here: The authors report strong generalization to deeper structures on COGS, challenging a literal claim that systems never master compositional tasks. This restricted semantic-parsing result does not establish broad compositional understanding.
- Publisher
- arXiv; Pia Weißenhorn and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
An inventory of the relevant claimed solutions and controlled evaluations covering their stated scope, or a pre-post demonstration of robust broad compositional understanding.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
But many people have claimed over the years to solve the problem, and none of those systems have proven to be reliable; every proposed solution has been like Clever Hans, working in dim light, leveraging large databases to some degree, but falling apart upon careful inspection; they might get 60% on some task, but they never really master it.
- Where
- paragraph 32
- Published
- 2022-09-14
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-did-googleai-just-snooker-one-of-8f381c-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-did-googleai-just-snooker-one-of-8f381c-a1.