“To our knowledge, no LLM-based system can yet reliably do what SHRLDU tried to do then:”
Our reading, the sentence that was checked:
As far as the authors know, no LLM-based system can yet reliably do the blocks-world modeling that SHRDLU attempted in the 1970s.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Hello, Multimodal Hallucinations, paragraph 18.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-hello-multimodal-hallucinations-1c3721-a1
We recorded 10 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-10-21. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- to our knowledge (qualifier, stated by the speaker)
From the speaker
- To our knowledge
- no LLM-based system
- can yet reliably do
- what SHRLDU tried to do then
Our interpretation
- SHRLDU is read as SHRDLU, the system identified in the surrounding text.
- The checkable proposition concerns publicly demonstrated capabilities, not the authors' private knowledge.
- The ordinary comparison covers consistent blocks-world understanding, state tracking and execution of language-directed manipulations across an exchange.
- LLM-based systems include systems that combine an LLM with an external planner; the claim does not explicitly exclude them.
- Reliably means repeatable success across varied relevant tasks, rather than a selected successful demonstration. No numerical cutoff is supplied.
- For the ongoing capability state, I use January 1–October 21, 2023, the current research reporting year through publication.
Counts as supported if: Broad contemporaneous evaluations covering relevant LLM-based systems, including hybrids, establish persistent failures across the SHRDLU-style task bundle and identify no reliable qualifying system.
Counts as contradicted if: A publicly documented system existing by October 21, 2023 repeatedly performs comparable blocks-world language understanding, state tracking and manipulation reliably.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Contemporaneous studies document weak autonomous planning, but successful LLM-plus-planner systems prevent a blanket inference about every LLM-based system. Those hybrid results also do not establish the full conversational and world-tracking comparison. The undefined reliability threshold and task scope materially affect the answer, so neither support nor contradiction is secure.
Limitations
- Planning benchmarks cover only part of the ordinary SHRDLU comparison.
- Failure to locate a complete qualifying system does not prove that none existed.
- The authors' private knowledge cannot be independently assessed.
Evidence (2 items)
Context only: arXiv; Karthik Valmeekam and colleagues,
show more promise
Why it matters here: The study reports limited autonomous planning performance, including an approximately 12% cross-domain average for GPT-4, alongside more promising assisted approaches. These results do not establish that every LLM-based system lacked the full claimed capability.
- Publisher
- arXiv; Karthik Valmeekam and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Bo Liu and colleagues,
an optimal plan
Why it matters here: LLM+P combines language-model translation with a classical planner and reports successful planning across benchmark domains. This is material counterevidence to excluding all LLM-based systems, but it does not establish reliable performance across the full conversational comparison.
- Publisher
- arXiv; Bo Liu and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Contemporaneous records of a qualifying system, or broad evaluations against a clearly specified SHRDLU-style task suite, including hybrid systems and repeated reliability measurements.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Hello, Multimodal Hallucinations
To our knowledge, no LLM-based system can yet reliably do what SHRLDU tried to do then:
- Where
- paragraph 18
- Published
- 2023-10-21
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-hello-multimodal-hallucinations-1c3721-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-hello-multimodal-hallucinations-1c3721-a1.