“much of what makes them scary is the assumption that OpenAI will fix hallucinations, something that the company has categorically failed to do.”
Our reading:
OpenAI has failed to fix hallucinations.
Assessment: Supported
What happened
Was it true for the period described?
Outcome: Supported
The evidence matches what they said.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
On the plain reading, the question is whether OpenAI had fixed hallucinations by March 2024. It had not. OpenAI's own GPT-4 page says the model 'still is not fully reliable (it “hallucinates” facts and makes reasoning errors)' and calls hallucination 'still a real issue'. A Stanford study published in January 2024 tested OpenAI's GPT-3.5 among other models and found hallucination rates of 69% to 88% on specific legal queries. OpenAI did report reducing hallucinations relative to earlier models. That cuts against reading 'categorically' as 'no progress at all', but not against the plain claim that OpenAI had failed to fix the problem.
Caveats
- The independent study covers GPT-3.5 and the legal domain, not GPT-4 in general.
- 'Categorically' is read as 'has not fixed'. Real reductions were reported.
Evidence (4 items)
Supports the claim: OpenAI,
it still is not fully reliable (it “hallucinates” facts and makes reasoning errors)
Why it matters: OpenAI itself said its flagship model still hallucinates facts. That model was still the flagship at the time of the post.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- About events on
- Published
Supports the claim: Stanford HAI / RegLab,
hallucination rates range from 69% to 88% in response to specific legal queries for state-of-the-art language models
Why it matters: An independent study two months before the post found high hallucination rates in models including OpenAI's GPT-3.5 (see e3).
- Publisher
- Stanford HAI / RegLab
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
Context only: Stanford HAI / RegLab,
We tested more than 200,000 queries against each of GPT 3.5, Llama 2, and PaLM 2
Why it matters: Confirms that an OpenAI model was among those tested in e2. GPT-4 was not tested.
- Publisher
- Stanford HAI / RegLab
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
Context only: OpenAI,
While still a real issue, GPT‑4 significantly reduces hallucinations relative to previous models
Why it matters: OpenAI reported real progress, which undercuts a 'no progress at all' reading of 'categorically'. It also calls hallucinations 'still a real issue', so the problem was not fixed.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- About events on
- Published
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Where we had to interpret
- Plain reading: as of the post, OpenAI had not stopped its models from hallucinating. They still made up facts at rates that matter in practice, as shown by OpenAI's own statements or published evaluations.
- 'Categorically failed' is read as 'has not fixed', not as 'made no progress at all'. Partial reduction does not contradict the claim unless hallucinations were close to eliminated.
- Target period: from ChatGPT's launch (2022-11-30) to the post date. This is the record of OpenAI's efforts that the claim refers to.
Right if: OpenAI's own documentation or independent evaluations from the period show its models still hallucinated in meaningful ways.
Wrong if: Evidence from the period shows OpenAI's models had eliminated or nearly eliminated hallucinations.
We wrote this test after the outcome was known.
We found 17 claims in this post. See them all.
Full record
Source and dates
Where's Your Ed At, Have We Reached Peak AI?, paragraph 29.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-28. The post may have been edited after 2024-03-18.
Where it was said (1)
Said once in the posts we read.
: Where's Your Ed At, Have We Reached Peak AI?
much of what makes them scary is the assumption that OpenAI will fix hallucinations, something that the company has categorically failed to do.
- Where
- paragraph 29
- Published
- 2024-03-18
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Supported. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.