“Forty one years later, neural networks (at least of the popular variety) still struggle with this. They still remain pointillistic masses of blurry memory, never as systematic as reasoning machines ought to be.”
Our reading, the sentence that was checked:
Popular neural networks still struggle with systematic relational reasoning and are not as systematic as reasoning machines should be.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Elegant and powerful new result that seriously undermines large language models, paragraph 11.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-elegant-and-powerful-new-result-that-6cd679-a2
We recorded 8 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-09-22. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- popular variety (qualifier, stated by the speaker)
From the speaker
- neural networks
- at least of the popular variety
- still struggle with this
- never as systematic as reasoning machines ought to be
Our interpretation
- Assess the empirical assertion that contemporary popular neural models have substantial failures of systematic relational and compositional reasoning.
- The metaphor about blurry memory and the standard expressed by ought are not separately measurable claims.
- Because model evaluations have no fixed quarterly reporting cycle, use the latest annual evaluation period available before the post: 2023, restricted to work preceding September 22.
Counts as supported if: Controlled evaluations of contemporary widely used neural models show substantial failures when combining or transferring learned relationships and reasoning operations.
Counts as contradicted if: Relevant controlled evaluations show consistently reliable systematic transfer, with the alleged failures absent or attributable to invalid tests.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The June 1, 2023 preprint evaluated GPT-3, ChatGPT and GPT-4 on multiplication, logic grid puzzles and dynamic programming, finding sharply declining performance as compositional complexity increased [e1]. These results support systematic reasoning limitations in those models and tasks [e1]. Anthropic's August 8, 2023 report found more abstract and stronger cross-language generalization with model size, limiting a literal pure-memorization reading of the metaphor [e2].
Limitations
- The tested models and tasks do not establish that every neural architecture lacks systematic reasoning [e1].
- The evidence does not establish that model outputs are merely memorized fragments [e2].
- The normative standard of how systematic reasoning machines ought to be is not measured by these studies [e1, e2].
Evidence (2 items)
Supports the claim: arXiv; Dziri et al.,
failures on surprisingly trivial problems
Why it matters here: The study evaluates GPT-3, ChatGPT and GPT-4 on multiplication, relational logic puzzles and dynamic programming. Performance deteriorates sharply as compositional complexity increases, supporting the claim that popular models struggle with systematic reasoning.
- Publisher
- arXiv; Dziri et al.
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: Anthropic,
stronger with model size
Why it matters here: The researchers find increasingly abstract and cross-language generalization as models grow. This limits a literal interpretation that neural models are only stores of memorized fragments; it does not negate demonstrated systematicity failures.
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Elegant and powerful new result that seriously undermines large language models
Forty one years later, neural networks (at least of the popular variety) still struggle with this. They still remain pointillistic masses of blurry memory, never as systematic as reasoning machines ought to be.
- Where
- paragraph 11
- Published
- 2023-09-22
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-elegant-and-powerful-new-result-that-6cd679-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-elegant-and-powerful-new-result-that-6cd679-a2.