Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Gary Marcus Marcus on AI Published

The latest AI scaling graph - and why it hardly makes sense

We recorded 5 claims from this post. 3 were checked against evidence. 2 can’t be checked. 9 more are held. Read the post at its source.

Claims from this post (5)

In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.

  1. Assessment: MixedSaid

    Observation

    “METR’s blog and its tweets (perhaps written by publicists or by generative AI rather than by scientists), however, dropped the careful qualifications and nuance of the technical paper and made claims that go far beyond anything that the study actually supports.”

    Our reading: METR's blog and tweets made claims far beyond what its study supports, dropping the technical paper's qualifications.

    No horizon Evidence as of

    Limitation: This assesses the identified announcements, not every METR tweet.

  2. Assessment: UnresolvedSaid

    Observation

    “And it would be absurd to say that o3 can do all tasks that humans can do in 1.7 hours or that GPT-3 could do anything that a person can do in 15 seconds”

    Our reading: o3 cannot do all tasks humans can do in 1.7 hours, and GPT-3 could not do everything a person can do in 15 seconds.

    No horizon Evidence as of

    Limitation: Unresolved reflects an evidence gap, not evidence that either model has universal capability.

  3. Assessment: SupportedSaid

    Observation

    “Computers can answer some questions reliably now, for example, and some not, and the graph tells us nothing about which is which or when any specific question will be solved.”

    Our reading: The METR graph tells us nothing about which questions computers can reliably answer or when specific questions will be solved.

    No horizon Evidence as of

    Limitation: The outcome depends on reading the claim as about specific questions.

  4. Assessment: Can’t be checkedSaid

    Observation

    “That giant leap seems especially unwarranted given that there has likely been a lot of recent data augmentation directed towards software benchmarks in particular (where this is feasible).”

    Our reading: There has likely been a lot of recent data augmentation directed at software benchmarks in particular (where feasible), which makes the leap from software results to tasks in general seem especially unwarranted.

    No horizon

    Scope: where this is feasible

  5. Assessment: Can’t be checkedSaid

    Recommendation or value judgment

    “But attempting to use the graph to make predictions about the capacities of future AI is misguided, and the fact that it went viral is a sign that in AI (as in so many other domains) people are repeating things that they want to believe, based on their alleged conclusions rather than their validity.”

    Our reading: Using the METR graph to predict future AI capabilities is misguided, and its virality reflects people repeating what they want to believe.

    No horizon

Held from this post (9)

These records did not pass every check, so nothing about them is published. Held is not a finding.