“Embrace that which actually delivers value to your lives, not the theoretical promises of any number of monotonous billionaires that want to turn every website into fuel for a machine that continually gets things wrong.”
Our reading, the sentence that was checked:
Generative AI systems trained on web content continually get things wrong.
Assessment: Supported Evidence as of
Where and when Ed Zitron wrote it
Where's Your Ed At, Sam Altman Is Full Of Shit, paragraph 24.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment wyea-sam-altman-is-full-of-shit-2d6d39-a3
We recorded 15 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-05-21. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- a machine that continually gets things wrong
- turn every website into fuel
Our interpretation
- I assess the embedded factual assertion about generative AI errors, not the surrounding advice about what readers should embrace.
- The plain reading of continually is recurring factual mistakes across repeated use, not that every answer is wrong.
- The relevant systems are contemporary general-purpose generative language models. No numerical error rate or claim of inevitable future failure is supplied.
- For this ongoing state, I use April 2024, the latest complete calendar month before the May 21 post, as the reporting snapshot. These models have no standardized quarterly reporting period.
- Errors include unsupported factual additions when summarizing supplied material and false answers to factual questions. These tasks must not be treated as having interchangeable error rates.
Counts as supported if: Contemporaneous evaluations document recurring factual errors across several widely used generative models, using repeated tests rather than a single anecdote.
Counts as contradicted if: Broad, reproducible evaluations show that factual errors are absent or confined to isolated exceptions across representative tasks and contemporary systems.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Vectara’s April 30, 2024 snapshot reports document-summarization hallucination rates of 2.5% for GPT 4 Turbo, 4.6% for Gemini 1.5 Pro and 7.4% for Claude 3 Opus [e1]. Most summaries from these models were scored as factually consistent: 97.5%, 95.4% and 92.6%, respectively [e1].
Limitations
- The automated summarization scores do not establish error rates for factual question answering or everyday generative AI use [e1].
- The snapshot date does not establish when individual tests were run [e1].
- The evidence does not establish that most answers are wrong or that errors will persist indefinitely [e1, e2].
Evidence (2 items)
Supports the claim: Vectara,
when summarizing a document.
Why it matters here: The dated snapshot reports hallucination rates of 2.5% for GPT 4 Turbo, 4.6% for Gemini 1.5 Pro and 7.4% for Claude 3 Opus on document summarization. These recurring errors support the ordinary reading, although the automated scoring and restricted task do not establish a universal error rate.
- Publisher
- Vectara
- Kind of source
- Benchmark result
- Describes events of
- Public from
Context only: Hong et al., arXiv,
not the correct answer
Why it matters here: The paper illustrates false factual answers from smaller Bloom and Llama-2 models while showing correct answers from larger counterparts. It corroborates the mechanism and model variation, but does not date the individual test executions.
- Publisher
- Hong et al., arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Where's Your Ed At, Sam Altman Is Full Of Shit
Embrace that which actually delivers value to your lives, not the theoretical promises of any number of monotonous billionaires that want to turn every website into fuel for a machine that continually gets things wrong.
- Where
- paragraph 24
- Published
- 2024-05-21
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment wyea-sam-altman-is-full-of-shit-2d6d39-a3: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
HeldBy this site
The final-text check failed: rationale_supported_by_evidence; overclaims: These results support reading “continually” as recurring errors across repeated use [e1]..
About assessment wyea-sam-altman-is-full-of-shit-2d6d39-a2, which was not published.