About that OpenAI “breakthrough”
We recorded 10 claims from this post. 3 were checked against evidence. 7 can’t be checked. 2 more are held. Read the post at its source.
Claims from this post (10)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Observation
“Me? I am pretty skeptical; I was only half-kidding when I wrote this:”
Our reading: Marcus is pretty skeptical about OpenAI's reported Q* breakthrough.
No horizon
Qualifier: pretty
Assessment: Can’t be checkedSaid
Observation
“Although there is some talk of OpenAI already trying to test the new technique out, it wouldn’t be realistic for them to have entirely changed the world in a matter of months.”
Our reading: It is not realistic that OpenAI's Q* technique has already entirely changed the world within a few months.
No horizon
Assessment: Can’t be checkedSaid
Observation
“Part of the joke is that I don’t actually think Bing is powered already by the alleged breakthrough.”
Our reading: Marcus does not think Bing is already powered by OpenAI's alleged Q* breakthrough.
No horizon
Qualifier: I don't actually think
Assessment: Can’t be checkedSaid
Observation
“OpenAI could in fact have a breakthrough that fundamentally changes the world.”
Our reading: It is possible that OpenAI has a breakthrough that fundamentally changes the world.
No horizon
Assessment: UnresolvedSaid
Observation
“But “breakthroughs” rarely turn to be general to live up to initial rosy expectations. Often advances work in some contexts, not otherwise.”
Our reading: Claimed AI breakthroughs rarely prove general enough to live up to initial expectations, often working only in some contexts.
No horizon Evidence as of
Limitation: The result depends on which advances and which expectations are counted.
Assessment: Can’t be checkedSaid
Forecast
“We are probably still many “breakthroughs” away from true level 5 driverless cars.”
Our reading: True level 5 driverless cars are probably still many breakthroughs away.
No horizon
Assessment: SupportedSaid
Observation
“AV’s still aren’t general enough that you can just plop down a car that was tuned in pilot studies in Menlo Park, SF and Arizona and expect it to drive gracefully and safely in Sicily or Mumbai.”
Our reading: Autonomous vehicles still are not general enough that you can just plop down a car tuned in pilot studies in Menlo Park, SF and Arizona and expect it to drive gracefully and safely in Sicily or Mumbai.
No horizon Evidence as of
Limitation: The evidence documents Waymo's own testing process; it does not establish every developer's capabilities [e1].
Assessment: SupportedSaid
Observation
“getting their algorithm to work for one object (which, cheesily, turned out to be a special Cube instrumented with sensors and LEDS, unlike a Rubik’s cube you’d buy in a store) in carefully controlled lab circumstances hardly guaranteed that the solution would work more broadly in the complex open-ended real world.”
Our reading: OpenAI's 2019 Rubik's cube robot result, achieved on a sensor-instrumented cube in controlled lab conditions, did not guarantee generalization to the open-ended real world.
No horizon Evidence as of
Limitation: The selected documentation does not establish the precise claim that one demonstrated cube contained both sensors and LEDs.
Assessment: Can’t be checkedSaid
Observation
“It’s all still an empirical question; I for one certainly don’t yet know enough about the details of Q* to judge with certainty.”
Our reading: Whether Q* pans out is still an empirical question, and Marcus himself does not yet know enough about the details of Q* to judge it with certainty.
No horizon
Scope: I for one
Assessment: Can’t be checkedSaid
Observation
“At least so far, Q* ain’t one.”
Our reading: At least so far, OpenAI's Q* is not among the problems Marcus is worried about.
No horizon
Qualifier: at least so far
Held from this post (2)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews reached different assessments. Our reading, not assessed: OpenAI's touted Rubik's cube robotics breakthrough went nowhere, and OpenAI quietly closed its robotics division a year or two later.
- The two reviews reached different assessments. Our reading, not assessed: Autonomous cars are still not reliable enough for prime time; no breakthrough has gotten them over the threshold.