“No system has been shown to be robust across all the variations (though specific ones vary across replications, perhaps in response to publicly shared examples).”
Our reading, the sentence that was checked:
No system has been shown to be robust across all the variations of the posted LLM failure examples under discussion (such as those from Mahabal and Fraser), though which specific examples fail varies across replications, perhaps in response to publicly shared examples.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, AGI versus “broad, shallow intelligence”, paragraph 16.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-agi-versus-broad-shallow-intelligence-7f12a6-a1
We recorded 14 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-01-13. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- the variations of the failure examples discussed, such as Mahabal's and Colin Fraser's (scope, stated by the speaker)
- specific examples vary across replications, perhaps in response to publicly shared examples (qualifier, stated by the speaker)
From the speaker
- No system has been shown
- robust across all the variations
- specific ones vary across replications
- perhaps in response to publicly shared examples
Our interpretation
- This concerns public demonstrations available by January 13, 2025, covering the Mahabal and Fraser examples under discussion.
- Robust means consistently solving each identified variation across repeated trials with the same identified system. A single successful answer is insufficient.
- The target period is cumulative through publication because 'has been shown' describes the demonstration record, not just experiments conducted that day.
- The suggested response to publicly shared examples is explicitly speculative and is not assessed as a firm causal assertion.
Counts as supported if: A sufficiently complete, dated evaluation record covering the discussed variations and relevant systems shows that no system demonstrated consistent success across the complete set. Replications also document variation in which examples fail.
Counts as contradicted if: A public demonstration available by January 13, 2025 identifies a system that consistently handles the complete discussed set across repeated trials.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The recovered research documents failures under prompt variations, but it does not establish Marcus's universal claim about the discussed examples. A pre-publication evaluation also reports that o1 nearly solved a related puzzle collection. That is relevant counterevidence, but not a demonstration of success across his complete set. The missing prompt inventory and incomplete demonstration records prevent a fair supported or contradicted judgment.
Limitations
- Search failure cannot establish that no qualifying demonstration existed.
- Related benchmark results cannot substitute for the specific variations under discussion.
- Later model improvements would not contradict this historical observation.
Evidence (2 items)
Context only: arXiv; Mirzadeh and colleagues,
Specifically, the performance of all models declines when only the numerical values in the question are altered in the GSM-Symbolic benchmark.
Why it matters here: The study documents sensitivity to variations in mathematical questions. It does not evaluate the complete Mahabal/Fraser prompt set or establish the absence of every successful demonstration.
- Where
- Abstract
- Publisher
- arXiv; Mirzadeh and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: cpldcpu / Tim, GitHub,
One notable exception is o1 which is close to solving all prompts.
Why it matters here: The archived evaluation reports substantial success on a related 13-prompt collection, with repeated sampling described in its methods. Near-complete success on this collection neither demonstrates nor rules out robustness across Marcus's complete set.
- Where
- Reasoning models
- Publisher
- cpldcpu / Tim, GitHub
- Kind of source
- Independent measurement
- Describes events of
- Public from
What would settle it
An archived inventory of the discussed prompts and contemporaneous replication records, either showing a system consistently succeeding across the complete set or establishing the claimed absence through a sufficiently comprehensive evaluation record.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, AGI versus “broad, shallow intelligence”
No system has been shown to be robust across all the variations (though specific ones vary across replications, perhaps in response to publicly shared examples).
- Where
- paragraph 16
- Published
- 2025-01-13
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-agi-versus-broad-shallow-intelligence-7f12a6-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-agi-versus-broad-shallow-intelligence-7f12a6-a1.