“They’re really skilled at predicting what happens next, and not so skilled at reasoning, not so skilled at sticking to facts, and not very good at sticking to instructions.”
Our reading:
LLMs are good at next-step prediction but poor at reasoning, sticking to facts, and following instructions.
Assessment: Unresolved
What happened
Was it true for the period described?
Outcome: Unresolved
We checked, and the evidence so far doesn’t settle it.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
The evidence shows uneven capabilities, but does not settle Marcus's full comparison. Reliability studies document failures under false or adversarial context. Dated October benchmark records also show substantial mathematical achievements, which weigh against a broadly dismissive reading of reasoning ability. No retrieved study establishes the claimed ordering between text prediction and all three other capabilities for the contemporary model population. These records are insufficient for supported, contradicted, or mixed on the full proposition.
Caveats
- The claim supplies neither a common scale nor a task distribution.
- Formal mathematics uses a tool-assisted evaluation setting.
- The reliability papers do not provide verified October experimental dates in the retrieved text.
- Evidence of uneven performance alone does not establish the particular imbalance asserted.
Evidence (3 items)
Context only: Linghao Meng and colleagues, arXiv,
successful recovery remains rare
Why it matters: PHRBench reports difficulty recovering from deliberately introduced false context across 18 models. This supports a specific reliability concern, but unknown experimental dates and the absence of a text-prediction comparison prevent using it to settle the October capability imbalance.
- Publisher
- Linghao Meng and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
Context only: Epoch AI,
complete Lean proofs or disproofs.
Why it matters: Epoch records October 3 benchmark solutions by GPT-6.1 Sol to problems numbered 1, 74, and 126, using formally checked proofs. These are material counterevidence to a broadly dismissive account of reasoning, but tool-assisted successes on selected problems do not measure the entire claimed capability imbalance.
- Publisher
- Epoch AI
- Kind of source
- Independent measurement
- About events on
- Published
Context only: Scientific Reports,
weakest overall reliability
Why it matters: The study documents substantial errors on adversarial and ambiguous prompts, while also reporting differences among models. The retrieved text does not date the experiments closely enough to treat them as October outcome evidence, and it does not compare these results with next-token prediction.
- Publisher
- Scientific Reports
- Kind of source
- Peer-reviewed paper
- About events on
- Published
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Where we had to interpret
- In context, prediction means producing plausible next tokens or text continuations, not forecasting real-world events.
- The plain reading asserts a capability imbalance: strong text prediction alongside substantial recurring weaknesses in reasoning, factual reliability, and instruction adherence.
- The claim does not assert that models can never reason or answer correctly.
- I use October 2026, through the post date, as the latest reporting month for the ongoing comparison.
- A prediction-based training objective alone does not establish superior measured prediction skill relative to the other capabilities.
Right if: Contemporaneous comparative evaluations establish strong text-prediction performance together with substantial recurring weaknesses across reasoning, factual reliability, and instruction adherence.
Wrong if: Representative contemporary evaluations establish robust capabilities incompatible with the asserted broad imbalance. Conflicting evidence on different components may justify a mixed outcome if those components are adequately measured.
We wrote this test after the outcome was known.
What would settle it
Dated evaluations of representative October 2026 models covering text prediction, reasoning, factual reliability, and instruction adherence, with an explicit and defensible interpretation of relative skill.
We found 32 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, Report from the Vatican, paragraph 27.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-11-09. The post may have been edited after 2026-10-31.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, Report from the Vatican
They’re really skilled at predicting what happens next, and not so skilled at reasoning, not so skilled at sticking to facts, and not very good at sticking to instructions.
- Where
- paragraph 27
- Published
- 2026-10-31
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Unresolved. Evidence up to 2026-11-09.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.