“Some core problems have lingered, particularly hallucinations, instability and occasional nonsensical errors.”
Our reading, the sentence that was checked:
Hallucinations, instability and occasional nonsensical errors remain unsolved in GPT-4 class LLMs.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Happy 1st Birthday, GPT-4, paragraph 6.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-happy-1st-birthday-gpt-4-2cc74c-a1
We recorded 7 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-03-16. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Some core problems have lingered
- hallucinations
- instability
- occasional nonsensical errors
Our interpretation
- The ordinary reading is that these failure modes continued to occur, not that performance never improved or that every model failed equally often.
- Hallucinations are unsupported or false assertions. Instability includes material changes in answers or behavior across prompts, service updates, or deployments.
- Nonsensical errors include plainly incoherent output and elementary mistakes inconsistent with a model's otherwise demonstrated abilities.
- The birthday retrospective supplies the preceding year as the reporting period. Evidence must concern models or services operating during that year.
Counts as supported if: Evaluations and operational records document continuing hallucinations, materially variable behavior, and occasional incoherent or elementary errors in GPT-4-class systems during the target period.
Counts as contradicted if: Comparable evidence establishes that the named failure modes had been eliminated by publication. Lower error rates or a repair to one incident alone would not contradict persistence.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Research during GPT-4's first year documented hallucinations, elementary mistakes, and substantial changes in behavior. A February 2024 provider incident also documented incoherent output. These support the modest claim that such problems continued to occur. The February bug was fixed, and none of these records establishes that improvements were absent or that the problems are permanently insoluble.
Limitations
- The research tests selected tasks rather than representative everyday usage.
- The incident record identifies ChatGPT and an infrastructure fault, without naming the affected model version.
- ‘Have lingered’ is assessed as recurring failure modes over the retrospective year, not as a claim that every cited defect remained active on the post date.
Evidence (4 items)
Supports the claim: Zhang and colleagues, arXiv,
extremely simple mistakes.
Why it matters here: The experiments demonstrate false supporting assertions that GPT-4 can recognize as wrong when asked separately. They establish hallucination and elementary-error failure modes, not a population-wide error rate.
- Publisher
- Zhang and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: Chen, Zaharia and Zou, arXiv,
vary greatly over time.
Why it matters here: Comparisons of March and June service versions found material changes across tasks. Some results improved, so this supports instability rather than uniform deterioration.
- Where
- Abstract, version 3
- Publisher
- Chen, Zaharia and Zou, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: OpenAI,
incorrect results
Why it matters here: OpenAI explains that a deployment bug produced word sequences that made no sense. This documents a contemporary service failure, not proof that the underlying model architecture necessarily produces nonsense.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: OpenAI,
ChatGPT is operating normally.
Why it matters here: The specific deployment incident was repaired before Marcus's post. It cannot establish that this particular bug remained active on March 16.
- Where
- Resolved update
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Happy 1st Birthday, GPT-4
Some core problems have lingered, particularly hallucinations, instability and occasional nonsensical errors.
- Where
- paragraph 6
- Published
- 2024-03-16
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-happy-1st-birthday-gpt-4-2cc74c-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.