“But GPT’s on their own don’t do scientific discovery.”
Our reading:
GPT models on their own do not do scientific discovery.
Assessment: Unresolved
What happened
Was it true for the period described?
Outcome: Unresolved
We checked, and the evidence so far doesn’t settle it.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
The March 22, 2023 preprint reports a correct GPT-4 proof, but does not establish an independently validated, scientifically novel discovery made by the model on its own [e2]. Adesso’s July 2023 account describes unsuccessful mathematical physics work with substantial human prompting, but cannot establish GPT capabilities generally through April 8, 2023 [e1]. These records establish neither a qualifying discovery by that cutoff nor a sufficiently broad assessment supporting the negative claim [e1] [e2].
Caveats
- The mathematical demonstrations do not establish the required novelty, autonomy and independent validation [e2].
- Adesso’s later account concerns a limited experiment and does not establish a precisely dated April 8 outcome [e1].
- The supplied evidence does not comprehensively assess contemporary GPT capabilities and purported discoveries [e1] [e2].
Evidence (2 items)
Context only: arXiv; Gerardo Adesso,
ChatGPT do science on its own? No, not yet.
Why it matters: The researcher reports substantial prompting and unsuccessful mathematical physics work in an experiment begun in December 2022. The July account does not establish the status of every GPT model by April 8, and its later revisions prevent treating this assessment as a precisely dated April outcome.
- Publisher
- arXiv; Gerardo Adesso
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
Context only: arXiv; Microsoft Research,
correct proof.
Why it matters: This records successful mathematical reasoning before the post. The nearby graph-theory demonstration also raises a novelty question, but its human-supplied construction and corrections do not establish an independently validated standalone discovery.
- Where
- Section 4.4, page 40
- Publisher
- arXiv; Microsoft Research
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
Probably right?
One AI thinks this is probably right, and the other says it’s too early to tell. The evidence isn’t conclusive, so the verdict above stays unresolved. It counts only when you pick Speculative under “How sure?”.
- Anthropic’s AI: A researcher who used ChatGPT on physics work said in 2023 it could not yet do science on its own. The March 2023 Microsoft preprint shows GPT-4 proving set problems with human help, not new findings. Nothing shows an independent discovery by April 2023.
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Conditions
- on their own (their words)
Where we had to interpret
- This is an observation about GPT models existing by April 8, 2023, not a forecast about subsequent models.
- The target is the state of demonstrated capability through the publication date, including April 1–8. No reporting quarter applies, and no lower date boundary is supplied.
- In context, GPT refers to OpenAI’s GPT family, including its ChatGPT implementations.
- The plain reading of scientific discovery is producing a substantively new scientific finding that withstands independent checking. A proposed hypothesis or scientific-looking text alone does not establish discovery.
- On their own permits a person to pose a research question and independently check the result. It excludes a person supplying the essential discovery or a larger research system supplying essential reasoning or experimental work.
Right if: Credible assessments covering contemporary GPT capabilities establish that standalone models had not demonstrated independently validated scientific discovery, with relevant purported discoveries examined. Isolated failures or an unsuccessful search alone are insufficient.
Wrong if: A documented finding generated by a GPT model by April 8, 2023 is independently established as scientifically novel and correct, with records showing that the model supplied the essential discovery under the stated autonomy criterion.
We wrote this test after the outcome was known.
What would settle it
A documented discovery generated by April 8, 2023, identifying the model, prompts, outputs, human and tool contributions, with independent validation of novelty and correctness; or a sufficiently broad historical assessment examining purported discoveries and establishing that standalone GPT models had not demonstrated this capability by that date.
We found 19 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, GPT-5 and irrational exuberance, paragraph 9.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-29. The post may have been edited after 2023-04-08.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, GPT-5 and irrational exuberance
But GPT’s on their own don’t do scientific discovery.
- Where
- paragraph 9
- Published
- 2023-04-08
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Unresolved. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
A reviewer said our test wasn’t fair to what was actually said.