“it’s even less clear how it would work with words like no and not (a major sore spot for current neural networks).”
Our reading, the sentence that was checked:
Handling negation words like 'no' and 'not' is a major weak spot for current neural networks.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Learning Language is Harder Than You Think, paragraph 16.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-learning-language-is-harder-than-50873c-a2
We recorded 7 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2022-07-20. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- words like no and not
- a major sore spot
- current neural networks
Our interpretation
- The language-acquisition context makes neural language models the relevant class of networks.
- A major sore spot means substantial, recurring difficulty interpreting negation. It does not mean every model fails whenever negation appears.
- Because the statement describes an ongoing research state without a reporting interval, the target period is the current research year through publication: January 1 to July 20, 2022.
- Comparisons must concern understanding negation, rather than merely detecting the presence of a negative word.
Counts as supported if: Contemporary evaluations show substantial, recurring errors when negation matters to the answer, with materially worse results than relevant affirmative or nonessential-negation comparisons.
Counts as contradicted if: Representative contemporary models handle meaning-changing negation reliably across varied tasks, with little meaningful disadvantage against comparable cases without negation.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Findings reported in March 2022 show substantial RoBERTa deficits on subsets where negation mattered: macro F1 was 0.67 versus 0.92 on QNLI and 0.73 versus 0.92 on QQP, comparing important with unimportant negation [e1](https://aclanthology.org/2022.acl-short.81.pdf). These results support a recurring negation weakness in the tested model and tasks, although the comparisons use different dataset subsets rather than matched sentences [e1](https://aclanthology.org/2022.acl-short.81.pdf). A May 2022 study reports more optimistic results from pragmatic tests of transformer language models, limiting how broadly this weakness can be generalized [e2](https://aclanthology.org/2022.acl-long.315/).
Limitations
- The supporting performance results concern RoBERTa and selected tasks; they do not establish a weakness in every contemporary neural network.
- The important- and unimportant-negation subsets are not matched sentence pairs, so their score differences do not isolate negation's causal effect.
- The evidence does not establish a population-wide negation penalty or uniform difficulty across linguistic contexts.
Evidence (2 items)
Supports the claim: Association for Computational Linguistics,
substantially lower with important negations.
Why it matters here: The study reports substantial RoBERTa performance losses on subsets where negation matters: QNLI macro F1 is 0.67 for important negations versus 0.92 for unimportant negations. These are dataset-subset comparisons, not matched sentence pairs; the findings were already reported in the March preprint.
- Publisher
- Association for Computational Linguistics
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Context only: Association for Computational Linguistics,
adhered to the semantic view.
Why it matters here: The authors report a substantially more optimistic assessment from pragmatic tests of transformer models. This is material counterevidence to a universal inability claim, but does not establish reliable performance across the tasks implicated by e1.
- Where
- Abstract
- Publisher
- Association for Computational Linguistics
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Learning Language is Harder Than You Think
it’s even less clear how it would work with words like no and not (a major sore spot for current neural networks).
- Where
- paragraph 16
- Published
- 2022-07-20
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-learning-language-is-harder-than-50873c-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-learning-language-is-harder-than-50873c-a2.