“Commonsense reasoning is still dodgy, as it has long been, as Ernie Davis and I just reviewed here..”
Our reading, the sentence that was checked:
Commonsense reasoning in current AI systems remains unreliable, as it long has been.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem, paragraph 12.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-sam-altman-thinks-that-agi-is-basically-6e0341-a1
We recorded 16 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-01-06. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Commonsense reasoning
- is still dodgy
- as it has long been
Our interpretation
- The ordinary reading is that contemporary general-purpose language models still make substantial, repeatable mistakes when reasoning about everyday situations. It does not mean they fail every commonsense task or have made no progress.
- The surrounding discussion identifies frontier language models as the principal subject.
- Because no reporting period is specified and benchmark evaluations are episodic, use the latest completed calendar year and the days through publication: 2024-01-01 through 2025-01-06. Earlier evaluations provide historical context for 'as it has long been'.
- No numerical reliability threshold is supplied. Assess whether systematic failures remain across multiple leading models on credible commonsense evaluations.
Counts as supported if: Contemporary evaluations demonstrate substantial, repeatable commonsense failures across leading language models, with earlier research documenting comparable limitations.
Counts as contradicted if: Broad contemporary evaluations establish consistently reliable commonsense reasoning across varied everyday situations, and the alleged failures are shown to be invalid, obsolete, or confined to unrepresentative systems.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Contemporary independent testing found substantial commonsense-dependent reasoning failures across several leading models, including o1-preview. Earlier PIQA results document that physical commonsense had also challenged preceding systems. Strong performance on some questions and large gains from tailored prompts qualify the claim, but do not remove the demonstrated reliability problem. This supports the ordinary reading of 'still dodgy'; it does not establish that models lack all commonsense or that progress had stopped.
Limitations
- Commonsense is broader than the tasks tested.
- The decisive contemporary study was a preprint.
- Task-specific prompting materially changes performance.
- The historical and contemporary benchmarks are not directly comparable.
Evidence (3 items)
Supports the claim: arXiv,
even the latest models lack the ability to effectively apply ToM reasoning
Why it matters here: SimpleToM tests everyday situations requiring inferences about what people can perceive and know. GPT-4o, Claude-3.5-Sonnet and Llama-3.1-405B scored only 10–24.9% on behavior judgments; o1-preview reached 59.5%. However, leading models exceeded 95% on mental-state questions, and tailored reminders plus reasoning prompts substantially improved applied reasoning.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv,
large pretrained models struggle (77%).
Why it matters here: PIQA documents an earlier gap in reasoning about ordinary physical interactions. It supports the historical background, but its older models and different benchmark cannot establish January 2025 performance or an unchanged error rate.
- Where
- Abstract
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv,
the o1 model has achieved the best performance on most datasets.
Why it matters here: The study documents improvements with o1, limiting any interpretation that reasoning had made no progress. Its commonsense category uses HotpotQA and constrained text generation, so it does not directly overturn SimpleToM's everyday social-reasoning failures.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Commonsense reasoning is still dodgy, as it has long been, as Ernie Davis and I just reviewed here..
- Where
- paragraph 12
- Published
- 2025-01-06
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-sam-altman-thinks-that-agi-is-basically-6e0341-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-sam-altman-thinks-that-agi-is-basically-6e0341-a1.