“What really matters is that the newest technology now supports writing stuff like this, at scale, and far more fluently than before.”
Our reading:
The newest language model technology now supports writing hoax content at scale and far more fluently than before.
Assessment: Unresolved
What happened
Was it true for the period described?
Outcome: Unresolved
We checked, and the evidence so far doesn’t settle it.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
The February 8, 2023 report supports rapid, repeated generation of readable false content within the February 1–12 target period [e1]. NewsGuard's January 2023 audit provides earlier context: ChatGPT generated false narratives for 80 of 100 tested narratives, sometimes requiring up to five attempts [e2]. Neither establishes a substantial fluency improvement over GPT-2 [e1] [e2]. A study available on December 24, 2022 found no statistically significant difference between GPT-3 and GPT-2 XL on a human–machine text-boundary detection task [e3]. That task did not directly measure fabricated-article fluency, so its result does not settle the claimed improvement; the full claim remains unresolved [e1] [e2] [e3].
Caveats
- The reports do not quantify sustained throughput or required human editing [e1] [e2].
- The January audit predates the target period [e2].
- The earlier comparison does not directly test target-period ChatGPT against GPT-2 on fabricated-article fluency, and its null result does not establish equal fluency [e3].
Evidence (3 items)
Supports the claim: The New York Times, reproduced on NowComment,
en masse
Why it matters: The contemporaneous report supports rapid, repeated production of fluent false content. Its own repeat experiment elicited disinformation on 33 percent of sampled questions, but it supplies no matched fluency comparison with GPT-2.
- Publisher
- The New York Times, reproduced on NowComment
- Kind of source
- News report
- About events on
- Published
Context only: NewsGuard,
false narratives
Why it matters: This original audit demonstrates repeated fabricated-content generation shortly before the target period. Some narratives required up to five attempts, and the audit did not compare fluency with GPT-2 or measure industrial throughput.
- Publisher
- NewsGuard
- Kind of source
- Independent measurement
- About events on
- Published
Context only: Liam Dugan and coauthors, arXiv,
mean annotator score
Why it matters: This earlier comparison cautions against assuming a large improvement on every writing-quality proxy. It measured detection of the boundary between human and generated text, not hoax-article fluency, and the authors say the task may need much more data to distinguish strong models.
- Publisher
- Liam Dugan and coauthors, arXiv
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
Probably right?
One AI thinks this is probably right, and the other says it’s too early to tell. The evidence isn’t conclusive, so the verdict above stays unresolved. It counts only when you pick Speculative under “How sure?”.
- Anthropic’s AI: NewsGuard's January 2023 audit had ChatGPT write false news articles for 80 of 100 narratives. The Feb 8 NYT report says it made clean variations en masse, unlike grammatically suspect predecessors. The Dugan study measured a different task, so it doesn't undercut this.
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Where we had to interpret
- The ordinary reading is that language models available by the post date could repeatedly generate readable fabricated articles with little human writing effort.
- The surrounding discussion identifies GPT-2 as the earlier comparison. It does not identify a particular GPT-2 size or configuration.
- Fluency means grammatical, readable, coherent prose. It does not mean factual accuracy.
- For the ongoing capability described as 'now', the target is the current month through the post date: February 1–12, 2023. There is no regular reporting quarter for this capability.
- The claim concerns production capability. It does not require proving that models produced the particular Bialek hoax or successfully manipulated search rankings.
Right if: Demonstrations or evaluations establish that models available in the target period could repeatedly produce plausible fabricated articles with little writing labor, with a substantial improvement in fluency over GPT-2.
Wrong if: Comparable demonstrations or evaluations show that the available models could not repeatedly produce such material with little writing labor, or that their fluency did not substantially improve over GPT-2.
We wrote this test after the outcome was known.
What would settle it
A reproducible comparison of models available during February 1–12, 2023 against specified GPT-2 configurations on comparable fabricated-article tasks, measuring readability, coherence, repeated generation and required human writing or editing.
We found 10 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, What Google Should Really Be Worried About, paragraph 20.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-29. The post may have been edited after 2023-02-12.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, What Google Should Really Be Worried About
What really matters is that the newest technology now supports writing stuff like this, at scale, and far more fluently than before.
- Where
- paragraph 20
- Published
- 2023-02-12
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Unresolved. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
The final check failed: a cited source didn’t say what the write-up said.