“Fundamentally, virtually any guardrail has to thread a needle between the Scylla of being too restrictive and Charybdis of being too permissive. None thus far have done this effectively.”
Our reading, the sentence that was checked:
No AI guardrail so far has effectively balanced being neither too restrictive nor too permissive.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight, paragraph 7.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-dall-es-new-guardrails-fast-furious-2b243e-a1
We recorded 8 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-01-10. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- thus far (qualifier, stated by the speaker)
From the speaker
- virtually any guardrail
- too restrictive
- too permissive
- None thus far have done this effectively
Our interpretation
- Assess guardrails available through January 10, 2024, especially safeguards for general-purpose language models discussed in the surrounding passage.
- Thus far is cumulative; no starting date is supplied, so the observation period has an open start.
- The ordinary reading of effective is a practically useful balance: reliably permit benign requests while resisting harmful requests, including adversarial attempts.
- Effective does not mean flawless. An isolated jailbreak or mistaken refusal is insufficient by itself to establish ineffectiveness.
- The universal claim requires broader evidence than selected demonstrations of failure.
Counts as supported if: Representative evaluations of the relevant guardrails show that none achieved a practically useful balance of benign-request acceptance and resistance to harmful requests.
Counts as contradicted if: At least one guardrail available by the post demonstrates that practical balance in sufficiently broad evaluations of both benign requests and adversarial harmful requests.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Pre-post research documents substantial guardrail failures, but it does not establish that none worked effectively. XSTest found GPT-4 performed well on both safe and unsafe prompts, and SmoothLLM demonstrated useful defenses with retained task performance. These are material counterexamples to an unqualified dismissal, but their test coverage does not establish broad jailbreak resistance with low over-refusal. Calling the claim supported or contradicted would require imposing an effectiveness threshold Marcus did not supply.
Limitations
- The cited evaluations cover selected models, prompts, and attacks.
- Useful task performance is not interchangeable with low over-refusal.
- Later research or updated model behavior cannot establish the state of guardrails at the post date.
Evidence (3 items)
Context only: Andy Zou and colleagues, arXiv,
objectionable behaviors
Why it matters here: The study demonstrates real failures of safeguards in multiple models. It does not measure benign-request refusals or establish that every guardrail lacked a practical balance.
- Publisher
- Andy Zou and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: Paul Röttger and colleagues, arXiv,
perform well
Why it matters here: XSTest found GPT-4 fully refused 6.4% and partially refused 2% of safe prompts, while refusing all but one of 200 unsafe prompts. This challenges a sweeping failure claim, but the unsafe contrasts were not a comprehensive adversarial safety evaluation.
- Publisher
- Paul Röttger and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: Alexander Robey and colleagues, arXiv,
high levels of nominal performance
Why it matters here: The paper reports substantial defense against GCG attacks while retaining useful performance. Its PAIR test still had an attack success rate around 50%, and general question-answering performance is not a direct measure of inappropriate refusal.
- Publisher
- Alexander Robey and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
A justified practical effectiveness standard and sufficiently broad, reproducible evaluations of both benign-request acceptance and adversarial safety for guardrails available by January 10, 2024.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight
Fundamentally, virtually any guardrail has to thread a needle between the Scylla of being too restrictive and Charybdis of being too permissive. None thus far have done this effectively.
- Where
- paragraph 7
- Published
- 2024-01-10
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-dall-es-new-guardrails-fast-furious-2b243e-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-dall-es-new-guardrails-fast-furious-2b243e-a1.