“Two years on, on the most important question of all – factuality and reliability — we are still pretty much where we were when ChatGPT first came out: wishing and hoping.”
Our reading, the sentence that was checked:
Two years after ChatGPT came out, on factuality and reliability, which he calls the most important question, we are still pretty much where we were at its release: wishing and hoping.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, ChatGPT, at Age Two, paragraph 40.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-chatgpt-at-age-two-352214-a2
We recorded 6 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-11-28. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- pretty much (qualifier, stated by the speaker)
- on factuality and reliability (scope, stated by the speaker)
From the speaker
- Two years on
- factuality and reliability
- still pretty much where we were when ChatGPT first came out
- wishing and hoping
Our interpretation
- The ordinary empirical reading is that factuality and reliability had improved little in substance since ChatGPT's launch.
- The comparison runs from the November 2022 launch to the November 2024 post.
- Calling factuality the most important question is a value judgment and is not graded.
- Continued hallucinations alone cannot establish little improvement.
- Pretty much supplies no numerical threshold. Treating this as a claim about unchanged error rates rather than continued need for verification materially affects the assessment.
Counts as supported if: Comparable evaluations of launch-era ChatGPT and November 2024 systems show little substantive improvement across relevant factuality and reliability tasks.
Counts as contradicted if: Comparable evaluations establish substantial improvements across relevant factuality and reliability tasks between those endpoints.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
A study published September 7, 2023 found fabricated citations in 55% of GPT-3.5 citations and 18% of GPT-4 citations on its literature-review tasks [e3]. A preprint revised November 7, 2024 reported factual precision of 90.8% for GPT-3.5-Turbo and 91.7% for GPT-4-Turbo, alongside larger gains in factual recall [e4]. Its GPT-3.5 comparator was a January 2024 snapshot [e4]. These studies show task-specific improvements and differing changes in precision and coverage, but neither supplies a matched comparison of launch-era ChatGPT and November 2024 systems across factuality and reliability [e3, e4]. They therefore do not settle whether improvement over the claimed period was substantively small [e3, e4].
Limitations
- Neither study establishes aggregate progress from November 2022 to November 2024 across representative factuality and reliability tasks.
- Citation fabrication, factual precision and factual coverage measure different aspects of performance.
- The evidence does not settle practical trustworthiness or define how much improvement would count as substantial.
Evidence (2 items)
Context only: Scientific Reports,
are fabricated.
Why it matters here: A paired study of literature reviews across 42 topics found substantially fewer fabricated citations with GPT-4. This weighs against a sweeping no-progress reading, but does not compare the actual November 2022 and November 2024 systems.
- Where
- Abstract
- Publisher
- Scientific Reports
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Context only: arXiv; Wei and colleagues,
factual precision
Why it matters here: On matched prompts, GPT-4-Turbo scored 95.0 versus GPT-3.5-Turbo's 79.6 on F1@64, while factual precision was 91.7% versus 90.8%. The different signals matter, and the GPT-3.5 model was the January 2024 version, not launch-day ChatGPT.
- Publisher
- arXiv; Wei and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Matched evaluations of an archived November 2022 ChatGPT baseline and November 2024 systems across representative tasks, reporting precision, coverage, abstention and serious errors separately, with an explicit standard for substantive improvement.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, ChatGPT, at Age Two
Two years on, on the most important question of all – factuality and reliability — we are still pretty much where we were when ChatGPT first came out: wishing and hoping.
- Where
- paragraph 40
- Published
- 2024-11-28
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-chatgpt-at-age-two-352214-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.