“But, as is evident from Eisenberg’s experiment, GPT-3 was not even able to generalize from moving a toy to moving a bookmark.”
Our reading:
GPT-3 failed to generalize the false-belief task from moving a toy to moving a bookmark.
Assessment: Unresolved
What happened
Was it true for the period described?
Outcome: Unresolved
We checked, and the evidence so far doesn’t settle it.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
The reported failure to generalize from moving a toy to moving a bookmark remains unverified. The packet supplies no independently accessible Eisenberg transcript or qualifying contemporary replication [marcus-how-not-to-test-gpt-3-4a0eb5-p1, stated gaps].
Caveats
- The packet contains no evidence items that establish whether the reported failure occurred.
- The paired prompts, responses, conversation history, model identity and test date remain unverified.
- The missing record prevents checking whether the response confused the expected bookmark location with the page where reading should resume.
Evidence (0 items)
Nothing we found settles this yet.
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Where we had to interpret
- The claim reports a failure in Eisenberg's particular bookmark variant, rather than asserting that no prompt could ever elicit a correct bookmark answer.
- Successful transfer means tracking the absent character's belief about the bookmark's location, separately from deciding where that character should resume reading.
- The experiment occurred before publication, but its exact date is unspecified. The target interval has an unknown start and ends on February 17, 2023.
Right if: An independent original transcript, or a faithful documented replication on the contemporary system, shows incorrect belief tracking in the bookmark variant despite success on the corresponding toy task.
Wrong if: The authenticated record shows correct belief tracking, or establishes that the alleged transfer failure resulted from a transcription error or a materially different question.
We wrote this test after the outcome was known.
What would settle it
An independently preserved transcript of Eisenberg's experiment from on or before February 17, 2023, or a faithful documented contemporary replication. It must include both tasks, complete conversation context, model identity and test date. The responses must distinguish the character's belief about the bookmark's location from where reading should resume.
We found 13 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, How Not to Test GPT-3, paragraph 25.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-29. The post may have been edited after 2023-02-17.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, How Not to Test GPT-3
But, as is evident from Eisenberg’s experiment, GPT-3 was not even able to generalize from moving a toy to moving a bookmark.
- Where
- paragraph 25
- Published
- 2023-02-17
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Unresolved. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
The final check failed: our summary didn’t match their words.