“Two and a half years later, more recent models no longer makes the same specific mistakes that GPT-3 did, but still lack a robust understanding of the real external world.”
Our reading, the sentence that was checked:
Recent LLMs no longer make GPT-3's specific mistakes but still lack a robust understanding of the real external world.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, AI still lacks “common” sense, 70 years later, paragraph 19.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-ai-still-lacks-common-sense-70-years-d51afb-a2
We recorded 15 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-01-05. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- more recent models
- no longer makes the same specific mistakes that GPT-3 did
- still lack a robust understanding of the real external world
Our interpretation
- Both the correction of the earlier examples and the continuing broader weakness must be assessed.
- The ordinary behavioral reading of robust understanding is dependable performance across varied physical and spatial problems.
- The surrounding references to contemporary models place the observation in late 2024 through the post date. I use 2024-10-01 through 2025-01-05.
- The phrase two and a half years later does not align with the passage's 2020 starting point and January 2025 publication.
Counts as supported if: Matched retests show that contemporary models reliably pass the cited GPT-3 failure cases, while varied physical and spatial evaluations show recurring basic failures.
Counts as contradicted if: Matched retests show that the cited earlier errors persist, or broad controlled evaluations establish dependable physical and spatial generalization in the contemporary models.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The PhysGame preprint, available December 2, 2024, reports average accuracy of 56.1% for GPT-4o and 55.2% for Gemini-1.5-pro on questions about physics violations in gameplay videos [e1]. This provides limited evidence of physical-reasoning weaknesses, but does not establish broad reliability across physical and spatial problems [e1]. It does not retest the specific earlier GPT-3 mistakes, so it cannot settle the claim that those errors were corrected while broader weaknesses remained [e1].
Limitations
- The packet contains no independent matched retests of the cited GPT-3 failure cases.
- PhysGame covers a narrower task than general understanding of the external world.
- The cited passages do not establish exact experimental dates within October 1, 2024 through January 5, 2025.
Evidence (1 item)
Context only: arXiv; Meng Cao and colleagues,
47.7% average accuracy.
Why it matters here: PhysGame reports physical-commonsense limitations across video language models, including GPT-4o and Gemini-1.5-pro. It supplies relevant behavioral context, but neither matched GPT-3 retests nor precise test dates establishing performance within the target interval.
- Publisher
- arXiv; Meng Cao and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Independent matched retests of the original GPT-3 examples using identified model versions from October 1, 2024 through January 5, 2025, alongside repeated, varied physical and spatial evaluations with documented prompts and sampling settings.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, AI still lacks “common” sense, 70 years later
Two and a half years later, more recent models no longer makes the same specific mistakes that GPT-3 did, but still lack a robust understanding of the real external world.
- Where
- paragraph 19
- Published
- 2025-01-05
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-ai-still-lacks-common-sense-70-years-d51afb-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-ai-still-lacks-common-sense-70-years-d51afb-a2.