“We have no way of guaranteeing that large language models won’t be jailbroken into saying unethical things. To the contrary, a just-published technique shows that current LLMs can be jailbroken quickly, and automatically.”
Our reading, the sentence that was checked:
There is no way to guarantee LLMs won't be jailbroken, and current LLMs can be jailbroken quickly and automatically.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, A CERN for AI and the Global Governance of AI, paragraph 11.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-a-cern-for-ai-and-the-global-governance-f505da-a1
We recorded 13 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-10-26. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- no way of guaranteeing
- large language models won’t be jailbroken
- saying unethical things
- a just-published technique
- current LLMs
- quickly, and automatically
Our interpretation
- The ordinary reading concerns general immunity from jailbreaks in useful, general-purpose LLMs, rather than guarantees limited to a specified attack family.
- Unethical things means harmful outputs that violate the tested models' safety restrictions.
- Quickly and automatically means an attack-generation process requiring no human intervention during its iterations and a modest number of queries or minutes. It does not imply success on every attempt or every model.
- I use October 1 through October 26, 2023, for the current research-state observation and the just-published technique.
Counts as supported if: An October 2023 publication demonstrates automated, relatively fast jailbreaks on multiple current LLMs, while examined defense guarantees do not establish general immunity.
Counts as contradicted if: The cited technique does not automate successful jailbreak discovery or does not operate quickly, or a contemporary defense establishes general immunity across the relevant jailbreak threat space.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
PAIR directly supports the claim about quick, automated jailbreaks across multiple contemporary LLMs. Its success varied markedly by model. Erase-and-check and SmoothLLM offered restricted guarantees, so Marcus's wording would be too broad if read as denying every kind of guarantee. Under the ordinary reading of general jailbreak immunity, those defenses do not contradict the claim.
Limitations
- The guarantee clause depends on the stated general-immunity interpretation.
- PAIR did not succeed on every model or harmful request.
- The experiments test specified harmful behaviors, not every possible meaning of unethical.
- The attack evidence comes from the authors' original preprint.
Evidence (3 items)
Supports the claim: arXiv; Patrick Chao and colleagues,
fully automated process
Why it matters here: The published demonstration reports roughly five-minute successful jailbreak generation and success rates of 60% for GPT-3.5, 62% for GPT-4, and 72% for PaLM-2. Success was substantially lower for Llama-2 and Claude, so the evidence establishes capability across multiple models rather than uniform vulnerability.
- Publisher
- arXiv; Patrick Chao and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Aounon Kumar and colleagues,
up to a certain size
Why it matters here: Erase-and-check is material counterevidence to a literal assertion that no guarantees of any kind existed. Its protection concerns bounded modifications and a safety filter, not immunity to arbitrary jailbreaks; it also predates the selected October window.
- Where
- Abstract
- Publisher
- arXiv; Aounon Kumar and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Alexander Robey and colleagues,
k-unstable
Why it matters here: SmoothLLM derives a high-probability defense guarantee under an assumption about how suffix attacks respond to character changes. This qualifies the sweeping wording, but does not establish general jailbreak immunity.
- Publisher
- arXiv; Alexander Robey and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, A CERN for AI and the Global Governance of AI
We have no way of guaranteeing that large language models won’t be jailbroken into saying unethical things. To the contrary, a just-published technique shows that current LLMs can be jailbroken quickly, and automatically.
- Where
- paragraph 11
- Published
- 2023-10-26
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-a-cern-for-ai-and-the-global-governance-f505da-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-a-cern-for-ai-and-the-global-governance-f505da-a1.