“And the thing that’s maybe most dangerous and that, I think, captures what all of us [speakers at this session] are saying here, is [current AI] doesn’t follow instructions very well.”
Our reading:
Current AI does not follow instructions very well, which is perhaps its most dangerous trait.
Assessment: Supported
What happened
Was it true for the period described?
Outcome: Supported
The evidence matches what they said.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
The October AgentIF results support the ordinary reliability claim: multiple current models repeatedly fail explicit constraints in realistic agent instructions. Full-instruction success is approximately 31% even for leading entries. This supports a substantial weakness, not an inability to follow instructions at all. Earlier research shows that familiar instruction tests can produce strong results. The proposed support therefore depends on reading Marcus as describing unreliable adherence, rather than uniformly poor performance on every task.
Caveats
- The qualitative threshold requires interpretation.
- Current evidence is strongest for complex agent instructions and open-weight models.
- The tentative claim about greatest danger is not assessed.
Evidence (2 items)
Supports the claim: AgentIF research team,
all constraints are satisfied.
Why it matters: The dated leaderboard reports DeepSeek-V4-Pro satisfying 67.44% of individual constraints and all constraints on 31.27% of instructions; Qwen3.8-27B scores 65.77% and 31.35%, respectively. Across the listed open-weight models, these results show recurring failures on instructions drawn from real agent applications, supporting the reliability reading of the claim.
- Publisher
- AgentIF research team
- Kind of source
- Benchmark result
- About events on
- Published
Context only: Pyatkin and colleagues, arXiv,
models display good accuracy on IFEval.
Why it matters: This study reports strong performance on familiar IFEval constraints alongside much weaker generalization to new constraints. It limits any interpretation that instruction following is uniformly poor, but its earlier model evaluations cannot settle October 2026 performance.
- Publisher
- Pyatkin and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Where we had to interpret
- The surrounding discussion identifies the target as general-purpose LLMs.
- The plain reading is that these systems have substantial, recurring difficulty reliably obeying clear, feasible instructions. It does not mean they fail every instruction.
- I use October 2026, through the post date, as the latest reporting month for this ongoing capability claim.
- The tentative ranking of this weakness as perhaps the most dangerous is not separately scored.
Right if: Contemporaneous evaluations across multiple models and realistic tasks show substantial recurring violations of explicit instructions.
Wrong if: Representative contemporaneous evaluations show consistently reliable adherence across realistic instruction types, with failures confined to exceptional cases.
We wrote this test after the outcome was known.
We found 32 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, Report from the Vatican, paragraph 23.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-11-09. The post may have been edited after 2026-10-31.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, Report from the Vatican
And the thing that’s maybe most dangerous and that, I think, captures what all of us [speakers at this session] are saying here, is [current AI] doesn’t follow instructions very well.
- Where
- paragraph 23
- Published
- 2026-10-31
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Supported. Evidence up to 2026-11-09.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.