“Compositionality has long been a central concept in linguistics and philosophy (even Aristotle acknowledged it), yet so-called foundation models (large language models, GPT-3, BERT, deep learning approaches) sidestep it.”
Our reading, the sentence that was checked:
Foundation models such as GPT-3 and BERT sidestep compositionality.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Three ideas from linguistics that everyone in AI should know, paragraph 41.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-three-ideas-from-linguistics-that-42e048-a2
We recorded 26 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2022-06-22. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- foundation models
- large language models, GPT-3, BERT, deep learning approaches
- sidestep it
- Compositionality
Our interpretation
- In the surrounding examples, the ordinary behavioral reading is that these models do not reliably preserve the meanings of parts and their arrangement when interpreting novel combinations.
- This reading does not equate the absence of a hand-coded semantic module with the absence of learned compositional ability.
- Sidestep could instead describe architecture or training objectives; those readings are recorded separately rather than silently substituted.
- Current models are assessed over the current calendar year through publication, 2022-01-01 to 2022-06-22, because model evaluations have no regular reporting period.
- The assessment addresses the normalized proposition about foundation models, not the historical aside about Aristotle.
Counts as supported if: Representative evaluations of contemporary foundation models establish systematic failure to preserve part meanings and their relations across novel combinations.
Counts as contradicted if: Representative evaluations of contemporary foundation models establish reliable compositional interpretation across controlled novel combinations. A narrow task success qualifies an absolute dismissal but does not alone establish general reliability.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
A May 2022 evaluation reported that all tested models failed its composition tasks in zero-shot conditions [e1]. Another May 2022 study reported 99.7% accuracy for code-davinci-002 on SCAN using least-to-most prompting, demonstration examples, Python notation and postprocessing [e2]. These results establish weaknesses without examples and success in one assisted setting, but do not settle compositional reliability across foundation models [e1, e2].
Limitations
- The studies use different tasks and assistance levels, preventing a direct comparison [e1, e2].
- Neither study establishes general reliability or general failure across the named models and compositional phenomena [e1, e2].
- These behavioral results do not settle an architectural interpretation of 'sidestep'.
Evidence (2 items)
Supports the claim: Lialin et al., ACL 2022 / arXiv,
fail on composition tasks
Why it matters here: The tested pretrained models showed important compositional weaknesses without task examples. The paper also reports strong improvements after fine-tuning, so the result supports a reliability concern rather than an absence of all compositional capacity.
- Publisher
- Lialin et al., ACL 2022 / arXiv
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Context only: Zhou et al., arXiv,
least-to-most prompting achieves an accuracy of 99.7%.
Why it matters here: A contemporary GPT-3 model performed strongly on SCAN's length generalization split using structured decomposition prompts, Python notation and postprocessing. This materially challenges an absolute dismissal, but the narrow, assisted setup does not settle general compositional reliability.
- Where
- Section 3.2, Results
- Publisher
- Zhou et al., arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Representative controlled evaluations of models available during January 1–June 22, 2022, testing held-out compositional combinations with specified model versions, prompting, fine-tuning and reliability criteria.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Three ideas from linguistics that everyone in AI should know
Compositionality has long been a central concept in linguistics and philosophy (even Aristotle acknowledged it), yet so-called foundation models (large language models, GPT-3, BERT, deep learning approaches) sidestep it.
- Where
- paragraph 41
- Published
- 2022-06-22
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-three-ideas-from-linguistics-that-42e048-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-three-ideas-from-linguistics-that-42e048-a2.