“I think it’s still very poor reasoning.”
Our reading:
Current LLMs are still very poor at reasoning.
Assessment: Contradicted
What happened
Was it true for the period described?
Outcome: Contradicted
The evidence goes against what they said.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
October 2026 evaluations contradict a general description of leading LLMs as very poor at reasoning [e1, e2, e3]. ARC Prize reported Gemini 4 Argon scoring 98.5% on semi-private ARC-AGI-1 and about 91% on ARC-AGI-2 at high reasoning on October 13 [e1]. Artificial Analysis reported Claude Opus 5.5 and GPT-6 Astra each scoring 78% on HLE-Diamond reasoning questions at maximum effort on October 27 [e2]. In First Proof's October 2–5 testing, the direct GPT-6 Astra test solved 7 of 10 research mathematics problems correctly, while the ProofCouncil harness solved 9 [e3]. Weaknesses remain: every submission from the open-weight mathematics entrant contained major errors, and ARCANA found analogy gaps in older model generations [e3, e4].
Caveats
- These results do not establish reliable reasoning in every real-world setting or a particular internal reasoning mechanism.
- Results depend on model generation, reasoning settings and, in some evaluations, tools or harnesses [e1, e2, e3].
- First Proof recorded remaining citation and exposition weaknesses [e3].
- ARCANA did not test the leading models in the cited October success reports [e4].
- The ARC-AGI-2 result differs slightly between the report's prose, 91.2%, and table, 91.3% [e1].
Evidence (4 items)
Goes against the claim: ARC Prize,
98.5%
Why it matters: High performance on the semi-private abstract-reasoning tasks contradicts a general description of very poor reasoning. The prose reports 91.2% for v2 while the verified table reports 91.3%; this small discrepancy does not affect that conclusion.
- Publisher
- ARC Prize
- Kind of source
- Benchmark result
- About events on
- Published
Goes against the claim: Artificial Analysis,
72%
Why it matters: The physics evaluation uses a private 40-question hard subset and a code-execution tool. The same report separately gives Claude Opus 5.5 and GPT-6 Astra 78% on HLE-Diamond's reasoning questions, providing evidence of competence across providers and task types.
- Publisher
- Artificial Analysis
- Kind of source
- Independent measurement
- About events on
- Published
Goes against the claim: First Proof,
solved correctly
Why it matters: Testing occurred October 2–5 without human input; expert grading found seven correct solutions from the direct GPT-6 Astra test and nine from ProofCouncil. The report also records major errors throughout the open-weight entrant's submissions and remaining citation and exposition weaknesses.
- Publisher
- First Proof
- Kind of source
- Independent measurement
- About events on
- Published
Context only: arXiv,
analogical reasoning
Why it matters: The paper reports a human-model gap on far analogical reasoning. Its tested models are GPT 5.2, Gemini 3 Pro, Claude Opus 4.5, and Grok 4.1 Fast, so it does not directly establish the October frontier's capabilities.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Where we had to interpret
- Read the grammatical shorthand as saying current LLMs are generally very poor at reasoning.
- The ordinary test is performance on substantive reasoning tasks, including unfamiliar mathematical and abstract problems, rather than the mere existence of some failures.
- A broad description of current technology must account for leading available LLMs. It is not interpreted as a claim solely about small models, older generations, or models without reasoning settings.
- Use October 2026, the month of the post, for current capabilities. No benchmark or numerical threshold is supplied.
- Model-plus-harness results are distinguished from direct model tests.
Right if: Contemporary independent evaluations show broadly poor performance by leading LLMs across varied substantive reasoning tasks, with substantial failures relative to appropriate task baselines.
Wrong if: Contemporary independent evaluations demonstrate substantial reasoning competence across multiple demanding task families and leading models, making very poor an inaccurate general description.
We wrote this test after the outcome was known.
We found 32 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, Report from the Vatican, paragraph 23.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-11-09. The post may have been edited after 2026-10-31.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, Report from the Vatican
I think it’s still very poor reasoning.
- Where
- paragraph 23
- Published
- 2026-10-31
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Contradicted. Evidence up to 2026-11-09.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.