“But the interesting thing to note here is that we humans can choose to interpret language compositionally, or not—this degree of cognitive flexibility is really quite impressive, but lacking in current large language models.”
Our reading, the sentence that was checked:
Current LLMs lack the human ability to choose whether to interpret language compositionally.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Three ideas from linguistics that everyone in AI should know, paragraph 42.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-three-ideas-from-linguistics-that-a607b6-a1
We recorded 26 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2022-06-22. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- choose to interpret language compositionally, or not
- this degree of cognitive flexibility
- lacking in current large language models
Our interpretation
- The ordinary reading concerns reliable switching between literal and idiomatic interpretations as context changes. It does not require conscious choice.
- The comparison with humans concerns flexibility, not perfect accuracy or merely recognizing that an expression is an idiom.
- Current refers to models available when the post appeared, including the GPT-3 and BERT models named in its context. The target is their state on 2022-06-22; these capabilities have no regular reporting period.
Counts as supported if: Representative tests of contemporaneous models show substantial difficulty switching between literal and idiomatic meanings of the same expressions, where people reliably make that switch.
Counts as contradicted if: Representative tests demonstrate reliable context-appropriate switching between literal and idiomatic meanings by contemporaneous models, sufficient to refute the claimed absence of this flexibility.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The evidence cuts against an absolute claim of no flexibility: BERT research reported accurate separation of literal and idiomatic uses. Other research found substantial difficulty interpreting metaphors with GPT-3. Neither test directly establishes the human-like switching ability described here. These earlier results are context, not outcome evidence for the target snapshot. A firm verdict would depend on treating classification success or metaphor errors as a substitute for the claimed ability.
Limitations
- The available tests use different tasks and model configurations.
- The claim supplies no numerical standard for the human degree of flexibility.
Evidence (2 items)
Context only: INCOMA Ltd.; ACL Anthology,
with high accuracy.
Why it matters here: This weighs against reading Marcus as claiming absolutely no context-sensitive ability. It is an earlier classification-probe experiment, rather than a direct demonstration of the full interpretive flexibility described in the claim.
- Publisher
- INCOMA Ltd.; ACL Anthology
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Context only: arXiv,
accuracy, counting literalized metaphors as incorrect
Why it matters here: The experiment documents difficulty interpreting metaphors in an earlier GPT-3 test. Metaphor interpretation does not directly measure switching between literal and idiomatic readings of the same expression.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
A dated, representative evaluation of the contemporaneous models on paired literal and idiomatic contexts, with interpretation accuracy and human results, or an independently verified replication using preserved model versions.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Three ideas from linguistics that everyone in AI should know
But the interesting thing to note here is that we humans can choose to interpret language compositionally, or not—this degree of cognitive flexibility is really quite impressive, but lacking in current large language models.
- Where
- paragraph 42
- Published
- 2022-06-22
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-three-ideas-from-linguistics-that-a607b6-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-three-ideas-from-linguistics-that-a607b6-a1.