How come GPT can seem so brilliant one minute and so breathtakingly dumb the next?
We recorded 5 claims from this post. 2 were checked against evidence. 3 can’t be checked. 3 more are held. Read the post at its source.
Claims from this post (5)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Observation
“GPT-3 has no idea how the world works (and on this Bender and I would agree);”
Our reading: GPT-3 does not understand how the world works.
No horizon
Assessment: Can’t be checkedSaid
Observation
“But it is mimic that knows not whereof it speaks.”
Our reading: GPT-3 is a mimic that does not understand what it is saying.
No horizon
Assessment: SupportedSaid
Causal claim
“GPT doesn’t talk randomly, because it’s pastiching things actual people said.”
Our reading: GPT's output is not random, because it recombines things real people have said.
No horizon Evidence as of
Limitation: The result depends on reading pastiching as learned recombination rather than literal copying.
Assessment: UnresolvedSaid
Causal claim
“When GPT sounds plausible, it is because every paraphrased bit that it pastes together is grounded in something that actual humans said, and there is often some vague (but often irrelevant) relationship between..”
Our reading: GPT sounds plausible because each piece it combines comes from something humans actually said, and the pieces are often loosely related.
No horizon Evidence as of
Limitation: The meanings of bit, paraphrase and grounded materially affect the test.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“It was just named a Best Tech Article of 2022, and I believe that is still incredibly relevant, even with all the advances of the last 8 months.”
Our reading: Marcus believes his essay 'Deep Learning is Hitting a Wall' is still incredibly relevant, even with all the advances of the last 8 months.
No horizon
Qualifier: I believe
Held from this post (3)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews reached different assessments. Our reading, not assessed: GPT does not know which entities properly go with which properties.
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: For now, a human is still needed to judge which of GPT's plausible fragments actually belong together.