GPT-4’s successes, and GPT-4’s failures
We recorded 12 claims from this post. 6 were checked against evidence. 6 can’t be checked. 5 more are held. Read the post at its source.
Claims from this post (12)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
Observation
“It can see (though we don’t have a lot of details on that yet); it does astonishingly well on a whole bunch of standardized tests, like LSATs, GREs, and SATs.”
Our reading: GPT-4 has vision capability and performs astonishingly well on standardized tests such as the LSAT, GRE, and SAT.
No horizon Evidence as of
Limitation: The evidence is OpenAI's own evaluation report, without independent replication in the packet [e1].
Assessment: UnresolvedSaid
Observation
“It doesn’t actually solve any of the core problems of truthfulness and reliability that I laid out in my infamous March 2022 essay Deep Learning is Hitting a Wall.”
Our reading: GPT-4 does not solve any of the core truthfulness and reliability problems identified in Marcus's March 2022 essay.
No horizon Evidence as of
Limitation: This is an evidence gap for the full assertion, not evidence that the problems were solved.
Assessment: UnresolvedSaid
Observation
“Alignment is still shaky; you still wouldn’t be able to use it reliably to guide robots or scientific discovery, the kinds of things that made me excited about A(G)I in the first place.”
Our reading: GPT-4's alignment remains shaky and it cannot be used reliably to guide robots or scientific discovery.
No horizon Evidence as of
Limitation: Neither missing launch-period demonstrations nor general language-model errors prove application-wide inability.
Assessment: Can’t be checkedSaid
Observation
“Although GPT-4 is clearly better than GPT-3 and 3.5, it is not so far as I can tell, qualitatively better, only quantitatively better; as noted above, the limits remain more or less the same.”
Our reading: So far as Marcus can tell, GPT-4, though clearly better than GPT-3 and 3.5, is only quantitatively and not qualitatively better, and its limits remain more or less the same.
No horizon Evidence as of
Limitation: This does not reject the checkability of narrower claims about particular limitations or benchmark improvements.
Assessment: Can’t be checkedSaid
Observation
“Massive scaling has not thus far lead to revolution.”
Our reading: As of March 2023, massive scaling has not led to a revolution in AI.
No horizon
Qualifier: thus far
Assessment: Can’t be checkedSaid
Observation
“The quantitative improvements may (or may not) have considerable commercial implications in particular domains, but it’s not the massive win for the scaling hypotheses that memes like this had forecast.”
Our reading: GPT-4's quantitative improvements may or may not have considerable commercial implications in particular domains, but GPT-4 is not the massive win for the scaling hypotheses that memes like the one he cites had forecast.
No horizon
Qualifier: may (or may not)
Assessment: UnresolvedSaid
Observation
“On its own, it still requires frequent, massive retraining to be current with the news; GPT 3.5 knew nothing of 2022; GPT-4 seems to know little of 2023.”
Our reading: On its own, GPT-4 still requires frequent massive retraining to stay current with the news.
No horizon Evidence as of
Limitation: A knowledge cutoff is not evidence of complete ignorance of every later event.
Assessment: SupportedSaid
Observation
“We don’t know how big it is; we don’t know what the architecture is, we don’t know how much energy was used; we don’t how many processors were used; we don’t know what it was trained on etc.”
Our reading: OpenAI has not disclosed GPT-4's size, architecture, energy use, processor count, or training data.
No horizon Evidence as of
Limitation: The supplied excerpts do not directly establish the disclosure status of every listed category, particularly training energy and processor count.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“It is a step backwards for science, because it sets a new precedent for pretending to be scientific while revealing absolutely nothing.”
Our reading: GPT-4's release is a step backwards for science because it sets a new precedent for pretending to be scientific while revealing absolutely nothing.
No horizon
Assessment: UnresolvedSaid
Causal claim
“Because of all that, we don’t know what explains its success, and we don’t know how to predict its failures:”
Our reading: Because GPT-4's size, architecture, energy use, processors, and training data have not been disclosed, we don't know what explains its success and we don't know how to predict its failures.
No horizon Evidence as of
Limitation: The strongest quantitative evidence comes from OpenAI's own evaluations.
Assessment: Can’t be checkedSaid
Rhetoric
“One more giant step for hype, but not necessarily a giant step for science, AGI, or humanity.”
Our reading: GPT-4 is a big step for hype but not necessarily for science, AGI, or humanity.
No horizon
Assessment: Can’t be checkedSaid
Causal claim
“and (b) it puts all of us in an extremely poor position to predict what GPT-4 consequences will be for society, if we have no idea of what is in the training set and no way of anticipating which problems it will work on and which it will not.”
Our reading: If we have no idea what is in GPT-4's training set and no way of anticipating which problems it will work on and which it will not, all of us are in an extremely poor position to predict what GPT-4's consequences will be for society.
No horizon
Only if: if we have no idea of what is in the training set
Held from this post (5)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: When GPT-4 is integrated in Bing with current search results, hallucination remains rampant.
- The two reviews read the claim differently
- The two reviews read the claim differently