About that Matt Shumer post that has nearly 50 million views
We recorded 4 claims from this post. 2 were checked against evidence. 2 can’t be checked. 6 more are held. Read the post at its source.
Claims from this post (4)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Rhetoric
“The picture he sells just isn’t realistic, however much people might wish it were true.”
Our reading: The picture Shumer presents isn't realistic, however much people might wish it were true.
No horizon
Qualifier: however much people might wish it were true
Assessment: SupportedSaid
Observation
“LLMs write code wicked fast, but some coders are starting to report burnout, and only relatively modest gains relative to that burnout, as a new story from Connie Loizos at Tech Crunch illustrates”
Our reading: LLMs write code very fast, but some coders report burnout and only modest gains compared with that burnout.
No horizon Evidence as of
Limitation: The evidence is a small set of reports: one interviewed engineer and one anonymous forum commenter. It does not show how common burnout is among coders.
Assessment: SupportedSaid
Observation
“The security of the autogenerated code is very much in question, as for example this article last week noted (precisely as this substack has long warned):”
Our reading: The security of AI-generated code is seriously in doubt.
No horizon Evidence as of
Limitation: One main study comes from a security vendor with a commercial interest in highlighting risk. The other is a preprint.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“A lot of people may have taken his post seriously, but they shouldn’t have.”
Our reading: People should not have taken Shumer's viral post about AI seriously.
No horizon
Held from this post (6)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: When Shumer describes AIs doing longer and longer tasks on METR's task-time benchmark, he does not mention that the benchmark's criterion is 50% correct rather than 100%, or that the benchmark covers only coding and not tasks in general.
- The two reviews reached different assessments. Our reading, not assessed: Shumer offers no data showing that the latest coding systems can write whole complex apps without errors.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: No current AI system comes close to reliably doing every five-hour task humans can do without error, and hallucinations and boneheaded errors remain common in everyday use.