“Without a sense of what it is in the training corpus, it’s almost impossible to conclude anything about what a machine truly understands and how general that is.”
Our reading, the sentence that was checked:
Without knowing the training corpus, one can hardly conclude what a machine understands or how general it is.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, No, Virginia, AGI is not imminent, paragraph 12.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-no-virginia-agi-is-not-imminent-447af8-a1
We recorded 17 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-08-20. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- training corpus undisclosed (qualifier, stated by the speaker)
From the speaker
- Without a sense of what it is in the training corpus
- almost impossible to conclude anything
- what a machine truly understands
- how general that is
Our interpretation
- The plain reading is a strong practical limitation on drawing reliable conclusions about understanding and transfer when training contents are unknown.
- I assess observable transfer to novel tasks rather than consciousness or subjective understanding.
- The post describes contemporary evaluation practice without a reporting interval. I use 2023 through August 20 as the current research period; this is an evaluator-selected year-to-date window.
- The undisclosed-corpus qualification defines the systems under discussion; this is not a conditional forecast.
Counts as supported if: Evidence shows that even carefully designed novel-task evaluations of undisclosed-corpus systems generally cannot support reliable conclusions about understanding or transfer.
Counts as contradicted if: Reproducible evaluations with demonstrated task novelty establish substantive, reliable conclusions about transfer without access to training contents.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Contemporary experiments weigh against the strongest reading that virtually no useful conclusions are possible: researchers measured transfer failures and demonstrated use of an unpublished formal system. But the controlled variants were not guaranteed unseen, while the unpublished-system study was narrow. These records do not securely settle the stronger claim about reliable conclusions concerning understanding and its generality.
Limitations
- A reading limited to whether any bounded behavioral conclusion is possible would weigh more strongly against Marcus.
- Neither study establishes human-like understanding.
Evidence (2 items)
Context only: arXiv; Zhaofeng Wu and colleagues,
underperforms on counterfactual variants
Why it matters here: The researchers measured transfer limits in closed models without pretraining details, showing that useful behavioral conclusions were possible. However, they explicitly designed less-common task variants rather than guaranteeing that every variant was unseen.
- Where
- Figure 1 caption
- Publisher
- arXiv; Zhaofeng Wu and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Gregor vom Scheidt,
previously unpublished formal system
Why it matters here: The study reports GPT-4 applying a newly supplied formal system, which weighs against treating training access as necessary for every inference about transfer. It is a narrow demonstration, and the paper records errors as well as successes.
- Publisher
- arXiv; Gregor vom Scheidt
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Replicated evaluations of fixed 2023 models on demonstrably new tasks, with controls sufficient to establish reliable transfer conclusions without training-corpus access.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, No, Virginia, AGI is not imminent
Without a sense of what it is in the training corpus, it’s almost impossible to conclude anything about what a machine truly understands and how general that is.
- Where
- paragraph 12
- Published
- 2023-08-20
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-no-virginia-agi-is-not-imminent-447af8-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-no-virginia-agi-is-not-imminent-447af8-a1.