“An LLM just spits back (sometimes in transmogrified form) what it has read.”
Our reading, the sentence that was checked:
An LLM just spits back what it has read, sometimes in transmogrified form.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, April fools bring May hallucinations, paragraph 8.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-april-fools-bring-may-hallucinations-053004-a1
We recorded 2 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-04-04. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- sometimes in transmogrified form (qualifier, stated by the speaker)
From the speaker
- An LLM just spits back
- sometimes in transmogrified form
- what it has read
Our interpretation
- The ordinary reading is that LLM output is limited to reproducing or reworking previously encountered material, rather than applying learned rules to genuinely new problems.
- Transmogrified allows paraphrasing and rearrangement; different wording alone is not a counterexample.
- Novelty must be established against relevant training material. Familiar benchmark success is insufficient.
- Use January 1 through April 4, 2025: the latest completed quarter plus the April context of the ongoing claim.
Counts as supported if: Controlled studies show that apparent solutions to new problems are explained by reproduction or reworking of encountered examples, with no demonstrated generalization beyond them.
Counts as contradicted if: A controlled study demonstrates contemporary LLMs solving genuinely new instances through learned rules, with training overlap and externally supplied solutions adequately excluded.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The evidence challenges a simple repetition account, but does not settle this broadly worded claim under the stated controls. A January study reports transfer to unseen post-training rules, while relying on pretrained models and verifier feedback. A March study documents memorization problems, which cannot establish that all output is reproduction. Earlier controlled research is contrary context, but falls outside the observation window. Neither a categorical endorsement nor a categorical rejection is justified by this record.
Limitations
- Unresolved does not mean that the evidence favors the claim.
- The phrase transmogrified form leaves substantial interpretive uncertainty.
- The available studies test particular tasks and training methods.
- The January and March dates identify documented findings; exact experiment execution dates are not supplied.
Evidence (3 items)
Context only: arXiv; Chu and colleagues,
RL consistently improves
Why it matters here: The study reports transfer to rules withheld from post-training, which challenges a simple repetition account. It does not establish absence from all pretraining, and its sequential process incorporates verifier feedback.
- Publisher
- arXiv; Chu and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Chen and colleagues,
memorization remains a significant challenge
Why it matters here: The authors report code-generation failures under variations of familiar problems. This supports the existence of memorization problems, but does not establish Marcus’s exclusive claim about everything an LLM produces.
- Publisher
- arXiv; Chen and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Song, Xu and Zhong,
infer the hidden rules
Why it matters here: The research combines controlled synthetic transformer experiments with tests of pretrained LLMs and reports generalization. It is material contrary context, but predates the target window, and the fully controlled synthetic model is not itself a deployed large language model.
- Publisher
- arXiv; Song, Xu and Zhong
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
A dated experiment on a contemporary LLM with sufficiently controlled training exposure, genuinely new test solutions, and separate results without external solution-bearing feedback, together with an explicit boundary between reworking and generalization.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, April fools bring May hallucinations
An LLM just spits back (sometimes in transmogrified form) what it has read.
- Where
- paragraph 8
- Published
- 2025-04-04
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-april-fools-bring-may-hallucinations-053004-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-april-fools-bring-may-hallucinations-053004-a1.