“In a system that can write emails, make appointments, sign checks, etc, the consequences of unreliability, discomprehension, and hallucination all escalate.”
Our reading, the sentence that was checked:
When LLMs can take actions like writing emails or signing checks, the consequences of their unreliability and hallucinations escalate.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, What Could Possibly Go Wrong with Sam Altman’s New Ambitions?, paragraph 18.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-what-could-possibly-go-wrong-with-5e2c56-a1
We recorded 23 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-02-09. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- in systems that take real-world actions (qualifier, stated by the speaker)
From the speaker
- a system that can write emails, make appointments, sign checks
- the consequences of unreliability, discomprehension, and hallucination all escalate
Our interpretation
- The ordinary reading is that executing an erroneous instruction can directly change external systems and cause harm beyond producing an erroneous answer.
- This concerns the consequences of errors, not a claim that tool access necessarily increases error frequency.
- The examples describe a class of action-taking systems; they do not require every listed action or every deployment to produce harm.
Counts as supported if: Controlled demonstrations or documented incidents trace an agent error through an executed action to damage or disruption that an unexecuted advisory answer would not itself cause.
Counts as contradicted if: Comparative evidence shows that executing the relevant errors does not increase their consequences, including where the agent can change external systems.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The causal mechanism is demonstrated. ToolEmu researchers reproduced an agent action that crashed a virtual machine. Replit confirms an agent deleted database data in a setup where development actions could affect production. Executing the errors therefore produced consequences beyond an incorrect answer. Recovery and permission boundaries can reduce those consequences; Replit reports that the deleted data was fully restored.
Limitations
- The evidence supports increased potential consequences, not inevitable harm from every action.
- The sandbox result is not a production loss.
- No population-wide effect size is established.
Evidence (2 items)
Supports the claim: arXiv,
a crash
Why it matters here: The researchers reproduced destructive execution in a real sandbox, rather than merely generating harmful text. The paper traces agent failures involving misinterpretation and unsafe assumptions into consequential tool actions; the exact experiment date is not reported.
- Publisher
- arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: Replit,
production application
Why it matters here: Replit confirms that its agent deleted database data and explains how development actions could affect the live application. It also reports full restoration and subsequently introduced separation of development and production databases.
- Publisher
- Replit
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, What Could Possibly Go Wrong with Sam Altman’s New Ambitions?
In a system that can write emails, make appointments, sign checks, etc, the consequences of unreliability, discomprehension, and hallucination all escalate.
- Where
- paragraph 18
- Published
- 2024-02-09
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-what-could-possibly-go-wrong-with-5e2c56-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-what-could-possibly-go-wrong-with-5e2c56-a1.