“LLMs are unruly beasts; nobody knows how to make them refrain 100% of time from insulting users, giving bad advice, or just plain making stuff up.”
Our reading, the sentence that was checked:
LLMs are unruly, and nobody knows how to make them refrain 100% of the time from insulting users, giving bad advice, or making stuff up.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, How come smart assistants have virtually no ability to converse, despite all the spectacular progress with large language models?, paragraph 19.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-how-come-smart-assistants-have-virtually-bc7efe-a1
We recorded 10 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2022-11-22. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- 100% of the time (qualifier, stated by the speaker)
From the speaker
- LLMs are unruly beasts
- nobody knows how
- refrain 100% of time
- insulting users
- giving bad advice
- making stuff up
Our interpretation
- The ordinary reading concerns useful, open-ended language systems and the publicly established methods available when the post appeared.
- A system that remains silent or returns only fixed approved text is outside the conversational capability discussed in the post.
- The claim concerns complete protection, not whether researchers could substantially reduce harmful or false outputs.
- For this ongoing research-state observation, I use the current year's public research record through the post: 2022-01-01 to 2022-11-22. This is an evaluator-selected reporting window, not a date range supplied by Marcus.
- Support means the documented research state treated complete protection as unsolved; it does not require proving what every person privately knew.
Counts as supported if: Contemporary accounts of leading protection methods document remaining harmful or fabricated outputs and treat reliable prevention as an open problem, with no established method providing all three protections universally.
Counts as contradicted if: A method known by the post date demonstrably guaranteed all three protections across the intended open-ended use of a useful LLM.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The contemporary public research record supports the ordinary reading. OpenAI called reliable refusal of unsafe instructions an open problem. Sparrow's additional safety and grounding methods still produced failures, and Galactica explicitly lacked truthfulness guarantees. These records distinguish substantial mitigation from the complete protection Marcus specified. They do not establish that such protection is impossible forever.
Limitations
- This is a historical assessment of established methods, not a mathematical impossibility theorem.
- The cited systems do not exhaust every possible model or private experiment.
- Substantial safety improvements are compatible with the claim's 100% threshold.
Evidence (3 items)
Supports the claim: OpenAI,
reliably
Why it matters here: OpenAI described reliable refusal of unsafe instructions as unsolved despite improvements from human-feedback training. Its limitations section also reports remaining toxic outputs and fabricated facts.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Supports the claim: DeepMind researchers via arXiv,
78% of the time
Why it matters here: Sparrow combined human feedback, explicit behavioral rules, and supporting evidence. The paper still reports incomplete factual support and rule violations under adversarial probing; these improvements did not provide universal protection.
- Where
- Abstract
- Publisher
- DeepMind researchers via arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: Papers with Code team at Meta AI,
no guarantees of truthful output
Why it matters here: The contemporaneous model card states that scientific training data did not prevent hallucination and explicitly provides no truthfulness guarantee. The quotation preserves the source's spelling.
- Publisher
- Papers with Code team at Meta AI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, How come smart assistants have virtually no ability to converse, despite all the spectacular progress with large language models?
LLMs are unruly beasts; nobody knows how to make them refrain 100% of time from insulting users, giving bad advice, or just plain making stuff up.
- Where
- paragraph 19
- Published
- 2022-11-22
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-how-come-smart-assistants-have-virtually-bc7efe-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-how-come-smart-assistants-have-virtually-bc7efe-a1.