“What one really needs, and no one yet knows how to build, is a system that can derive semantics of wholes from their parts as a function of their syntax.”
Our reading:
No one yet knows how to build a system that derives semantics of wholes from parts as a function of syntax, which is what is needed.
Assessment: Unresolved
What happened
Was it true for the period described?
Outcome: Unresolved
We checked, and the evidence so far doesn’t settle it.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
October 2020 COGS experiments found weak and variable compositional generalization in the tested neural models [e1]. February 2022 research found that two compositional parsers maintained high accuracy across the tested COGS recursion depths [e2]. These results establish specific failures and successes, but neither a field-wide absence of reliable compositional understanding nor a method demonstrating that ability across varied ordinary language [e1, e2]. February 2020 research also identified distinct aspects of compositionality requiring separate tests [e3]. Together, these studies leave the field-wide claim about May 28, 2022 unresolved [e1, e2, e3].
Caveats
- The studies do not comprehensively assess methods available on May 28, 2022 [e1, e2, e3].
- The COGS results do not establish broad reliability across ordinary language, unfamiliar combinations, and sensible paraphrases [e1, e2].
Evidence (3 items)
Context only: arXiv; Najoung Kim and Tal Linzen,
generalization accuracy
Why it matters: The experiments document unstable compositional generalization in the tested architectures. They concern particular models and an earlier period, so they cannot establish Marcus's field-wide assertion.
- Publisher
- arXiv; Najoung Kim and Tal Linzen
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
Context only: arXiv; Pia Weißenhorn, Yuekun Yao, Lucia Donatelli and Alexander Koller,
high accuracy across all recursion depths.
Why it matters: This demonstrates successful structural generalization across the tested COGS recursion depths. The paper also discusses broad-coverage parsing, making it material counterevidence to a blanket dismissal of compositional systems, but these evaluations do not establish general reliability across ordinary language.
- Publisher
- arXiv; Pia Weißenhorn, Yuekun Yao, Lucia Donatelli and Alexander Koller
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
Context only: arXiv; Dieuwke Hupkes, Verna Dankers, Mathijs Mul and Elia Bruni,
different interpretations
Why it matters: The research treats compositionality as requiring several distinct tests. It explains why a single successful or failed benchmark does not settle the broader contextual claim.
- Publisher
- arXiv; Dieuwke Hupkes, Verna Dankers, Mathijs Mul and Elia Bruni
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Where we had to interpret
- In context, the ordinary reading concerns general, reliable compositional language understanding of the kind needed by systems such as Imagen. It does not deny that restricted semantic parsers can compose meanings.
- Relevant abilities include understanding unfamiliar combinations, distinguishing reversed participant roles, and preserving meaning across sensible paraphrases. These are contextual illustrations, not an added numerical benchmark.
- The observation describes the field's state on May 28, 2022. I use that date as the reporting snapshot; earlier experiments provide background rather than independently establishing the entire field's state that day.
- I assess the factual assertion about what researchers knew how to build. The phrase about what is needed supplies its practical context, rather than a separate claim that one particular architecture is necessary.
Right if: A sufficiently comprehensive assessment of methods available by the post establishes that general, reliable composition of sentence meanings from parts and syntax remained unsolved. Failures in selected systems alone do not establish that nobody knew how to build such a system.
Wrong if: A documented method available by May 28, 2022 demonstrates reliable, syntax-sensitive composition across varied ordinary language, including unfamiliar combinations and sensible paraphrases. Success confined to a restricted grammar or benchmark is insufficient by itself.
We wrote this test after the outcome was known.
What would settle it
A documented method available by May 28, 2022 demonstrating reliable, syntax-sensitive understanding across varied ordinary language, or a comprehensive historical assessment establishing that this capability remained unsolved then.
We found 13 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, Horse rides astronaut, paragraph 91.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-29. The post may have been edited after 2022-05-28.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, Horse rides astronaut
What one really needs, and no one yet knows how to build, is a system that can derive semantics of wholes from their parts as a function of their syntax.
- Where
- paragraph 91
- Published
- 2022-05-28
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Unresolved. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
A reviewer said our test wasn’t fair to what was actually said.