“Sometimes that works as a fine substitute, sometimes it doesn’t. You can never know in advance which.”
Our reading, the sentence that was checked:
LLMs' aping of what a person might say sometimes works as a fine substitute for what a person would do and sometimes doesn't, and you can never know in advance which.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, LLMs are not like you and me—and never will be., paragraph 36.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-llms-are-not-like-you-and-meand-never-c2a2fa-a1
We recorded 11 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-08-12. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- LLM imitation used as a substitute for human action (scope, stated by the speaker)
From the speaker
- Sometimes that works as a fine substitute
- sometimes it doesn’t
- You can never know in advance which
Our interpretation
- The claim concerns practical substitution for human task performance, not whether LLMs have human cognition.
- The plain reading is that users cannot dependably distinguish successful from unsuccessful substitutions before the particular attempt. This does not require philosophical certainty, but a weak predictive association is insufficient to contradict it.
- The observation concerns capabilities at publication. With no specified reporting interval for research results, I use the current reporting year, 2025, restricted to findings available by August 12. Earlier studies provide context.
- A task succeeds when it meets independently specified requirements. Answer checking after execution does not count as advance prediction.
Counts as supported if: Evaluations show both successful and unsuccessful task performance. Supporting the additional categorical claim requires evidence that dependable advance identification is unavailable across the claimed scope; failure of a few prediction methods alone is insufficient.
Counts as contradicted if: A method validated on previously unseen cases dependably identifies successful or unsuccessful substitutions before execution within a clearly defined task class. Merely predicting average performance, or checking completed answers, is insufficient.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The evidence supports variable success and substantial limits on advance prediction. It does not establish that users can never identify successful substitutions beforehand. Earlier research also shows that advance confidence can contain useful information, although it transfers imperfectly to unfamiliar tasks. These findings do not settle whether a method meets the ordinary standard of dependable advance identification across the work Marcus discusses. Scoring the whole claim as supported would turn limited experiments into a universal conclusion; scoring it as contradicted would treat partial predictability as sufficient without establishing that it is dependable.
Limitations
- The principal evaluation uses question-answering benchmarks rather than complete human workflows.
- The observation period and meaning of dependable identification require interpretation.
- The accessible studies do not exhaust possible prediction methods.
Evidence (2 items)
Supports the claim: Association for Computational Linguistics; author version on arXiv,
At higher thresholds, PVR drops to near zero for all assessors.
Why it matters here: PredictaBoard measures successes and failures and tests predictors that operate before the subject LLM receives an input. Its tested predictors leave almost no usable region above the discussed 95% accuracy threshold, supporting limited predictability in this setting without establishing universal impossibility.
- Publisher
- Association for Computational Linguistics; author version on arXiv
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Context only: Anthropic,
Models perform well at predicting P(IK) and partially generalize across tasks, though they struggle with calibration of P(IK) on new tasks.
Why it matters here: The study predicts whether a model knows an answer without inspecting a proposed answer, showing that advance information is possible. It predates the target period and does not establish dependable prediction of human-task substitution across unfamiliar settings.
- Where
- Abstract, sentence 7
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
What would settle it
A verified prospective study of models available at publication, using representative substitution tasks, predefined success requirements, and reported prediction errors and coverage, could establish a dependable counterexample. Supporting the categorical impossibility would require substantially broader evidence than failures of particular methods.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, LLMs are not like you and me—and never will be.
Sometimes that works as a fine substitute, sometimes it doesn’t. You can never know in advance which.
- Where
- paragraph 36
- Published
- 2025-08-12
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-llms-are-not-like-you-and-meand-never-c2a2fa-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-llms-are-not-like-you-and-meand-never-c2a2fa-a1.