“Nobody knows exactly how LLMs work, and that means that nobody can ever issue strong guarantees around them.”
Our reading:
Because nobody knows exactly how LLMs work, nobody can ever issue strong guarantees about them.
Assessment: Unresolved
What happened
Does the evidence support the cause?
Outcome: Unresolved
We checked, and the evidence so far doesn’t settle it.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
Anthropic's October 5, 2023 research documents limits of neuron-level interpretation, but does not establish that nobody understands any LLM [e1]. The June 3, 2024 alignment result establishes vulnerabilities under specified assumptions, without showing that incomplete understanding causes the claimed impossibility [e2]. CaMeL's June 24, 2025 version describes security protections enforced by surrounding software, with explicit exclusions and trust assumptions [e3]. CONTINUITY's September 4, 2026 version protects authenticated security facts and authorization decisions, but can preserve incorrect conclusions from trusted validators [e4]. These results neither establish the claimed causal barrier nor demonstrate the broad protection against open-ended attacks required to contradict it [e1, e2, e3, e4].
Caveats
- The interpretability evidence does not establish universal ignorance of LLM behavior [e1].
- The vulnerability result depends on assumptions and does not establish impossibility caused by incomplete understanding [e2].
- CaMeL excludes some manipulation attacks, and CONTINUITY depends on trusted validators; neither establishes the required broad security guarantee [e3, e4].
Evidence (4 items)
Context only: Anthropic,
activations of individual neurons
Why it matters: The research documents limits of interpreting individual neurons and a method for recovering interpretable features. It supports the narrower concern about incomplete understanding, but neither establishes that nobody understands any LLM nor proves that security guarantees are impossible.
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- About events on
- Published
Context only: Yotam Wolf and colleagues, arXiv,
Our framework makes several underlying assumptions.
Why it matters: The paper proves vulnerability to adversarial prompting under assumptions about distinguishable desirable and undesirable behavior components. This is a relevant limitation result, but its assumptions do not establish that incomplete human understanding prevents all strong system-level security guarantees.
- Publisher
- Yotam Wolf and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
Context only: Edoardo Debenedetti and colleagues, arXiv,
unintended actions and data exfiltration
Why it matters: CaMeL provides material counterevidence by enforcing security through system design around vulnerable models. However, it assumes trusted user prompts and uncompromised memory, and excludes text-only manipulation and phishing that leave protected flows unchanged; it therefore does not settle the broader claim under this contract.
- Publisher
- Edoardo Debenedetti and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
Context only: Chris Zheng and Geng Yang, arXiv,
authenticated security facts
Why it matters: This work extends formal protection across agent components, challenging the idea that security assurances must depend on understanding model internals. Its guarantee still depends on trusted validators and can preserve an incorrect conclusion, so it does not establish broad resistance to the open-ended manipulation discussed in the post.
- Publisher
- Chris Zheng and Geng Yang, arXiv
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
How we judged it
- Kind of claim
- Causal claim
- Deadline
- None given. We don’t add one.
Conditions
- because nobody knows exactly how LLMs work (their words)
Where we had to interpret
- In this passage about jailbreaks, private information and agents with device access, 'strong guarantees' means dependable security assurances against open-ended adversarial inputs, including previously unknown attacks.
- The claim concerns useful LLM deployments. An isolated guarantee about a restricted attack class does not by itself contradict it. Conversely, a relevant system-level guarantee is not excluded merely because external controls enforce it.
- 'Knows exactly how LLMs work' means a comprehensive understanding of their learned behavior, rather than knowledge of their architecture or execution rules.
- The central assertion is that incomplete understanding makes strong security guarantees impossible. This is assessed as a causal impossibility claim, not as a forecast with an invented deadline.
- Strong does not necessarily mean perfect protection against every conceivable risk. Marcus specifies neither a failure probability nor a precise security boundary.
Right if: Technical evidence establishes that incomplete understanding itself prevents dependable, broadly applicable security guarantees against open-ended jailbreaks in useful LLM deployments. Existing vulnerabilities or the absence of a comprehensive guarantee alone are insufficient.
Wrong if: Technical evidence establishes a dependable, broadly applicable security guarantee for a useful LLM deployment against open-ended adversarial inputs despite incomplete understanding of the model. Its protection must address the security concerns in the passage; a restricted perturbation certificate, isolated property or favorable benchmark score alone is insufficient.
We wrote this test after the outcome was known.
What would settle it
A rigorous result showing that incomplete model understanding itself prevents dependable security guarantees across the claim's scope would support it. A documented guarantee protecting a useful LLM deployment against open-ended adversarial inputs despite incomplete understanding would contradict it. Either result must state its security boundary, assumptions, residual risks, and retained usefulness.
We found 7 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, An unending array of jailbreaking attacks could be the death of LLMs, paragraph 6.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-29. The post may have been edited after 2024-04-02.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, An unending array of jailbreaking attacks could be the death of LLMs
Nobody knows exactly how LLMs work, and that means that nobody can ever issue strong guarantees around them.
- Where
- paragraph 6
- Published
- 2024-04-02
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Unresolved. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
The final check failed: the write-up said more than the evidence showed.
Held backBy us
A reviewer said our test wasn’t fair to what was actually said.