“These companies are building these models to excel at benchmarks because they can't train them to excel at defined tasks with any reliability”
Our reading, the sentence that was checked:
AI labs optimize for benchmarks because they cannot reliably train models to excel at defined tasks.
Assessment: Unresolved Evidence as of
Where and when Ed Zitron wrote it
Where's Your Ed At, The AI Hater's Manifesto, paragraph 83.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment wyea-the-ai-haters-manifesto-ab08b3-a2
We recorded 49 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2026-08-25. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- These companies
- building these models to excel at benchmarks
- because they can't train them
- to excel at defined tasks with any reliability
Our interpretation
- These companies refers to Anthropic, OpenAI and Meta in the surrounding passage.
- The claim asserts both a limitation in reliable task training and that this limitation causes benchmark-focused development.
- Reliable task performance means repeatable success on specified tasks outside the training examples, under stated operating conditions. No numerical reliability threshold is supplied.
- The assessment concerns training practice at the post date. Earlier research is relevant context but cannot alone establish current practice or motives.
Counts as supported if: Training research and decision evidence establish that inability to achieve reliable task performance materially drives benchmark-focused development at the named labs.
Counts as contradicted if: Robust evidence defeats the asserted inability in the relevant task scope, or direct causal evidence establishes that the alleged limitation does not explain benchmark-focused development.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Anthropic's July 2026 robotics report describes task successes and failures that depend on the control interface [e1](https://www.anthropic.com/research/claude-plays-robotics). Its November 2025 training study reports that one mitigation stopped misaligned generalization while leaving reward hacking unchanged [e2](https://www.anthropic.com/research/emergent-misalignment-reward-hacking). These limited experiments neither establish nor refute the claimed broad inability to train reliable models, and neither documents why the named labs prioritize benchmarks [e1](https://www.anthropic.com/research/claude-plays-robotics) [e2](https://www.anthropic.com/research/emergent-misalignment-reward-hacking).
Limitations
- Both studies predate the August 1–25, 2026 assessment period.
- The packet lacks comparable task evaluations across Anthropic, OpenAI and Meta during that period.
- Neither study connects reliability limitations to benchmark-focused development decisions.
- The claim specifies neither a task scope nor a numerical reliability threshold.
Evidence (2 items)
Context only: Anthropic,
Today’s frontier models cannot control humanoid robots without a pretrained policy
Why it matters here: The study documents both progress and persistent limitations on defined robotics tasks. Its domain and period are narrower than the claim, and it does not identify why labs optimize benchmarks.
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: Anthropic,
Fortunately, we found some mitigations that are effective.
Why it matters here: The study reports reward-hacking failures and some successful mitigations in a deliberately vulnerable training setup. It does not establish universal inability to train reliably or the alleged causal link to benchmark optimization.
- Where
- Mitigations
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
What would settle it
Repeated held-out task evaluations and documented training methods covering the named labs during August 1–25, 2026, together with decision records establishing whether reliability limitations materially drove benchmark-focused development. Robust evidence defeating the asserted inability, or direct evidence defeating the causal connection, could also settle the claim.
Where it was said (1)
Said once in the sources this register covers.
: Where's Your Ed At, The AI Hater's Manifesto
These companies are building these models to excel at benchmarks because they can't train them to excel at defined tasks with any reliability
- Where
- paragraph 83
- Published
- 2026-08-25
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment wyea-the-ai-haters-manifesto-ab08b3-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
HeldBy this site
The final-text check failed: citations_correct.
About assessment wyea-the-ai-haters-manifesto-ab08b3-a1, which was not published.