OpenAI’s dirty December o3 demo doesn’t readily replicate
We recorded 5 claims from this post. 5 can’t be checked. 6 more are held. Read the post at its source.
Claims from this post (5)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Observation
“And the lower 56% is not much different from what Jacob Andreas’s lab at MIT got in November. It’s arguably worse; if I followed correctly, and if the measures are the same, Andreas lab’s best score was actually higher, at 61%.”
Our reading: The 56% is not much different from what Jacob Andreas's lab at MIT got in November, and arguably worse, since, if Marcus followed correctly and the measures are the same, the Andreas lab's best score was higher at 61%.
No horizon Evidence as of
Limitation: This disposition concerns the qualified comparison, not the existence of the November result.
Assessment: Can’t be checkedSaid
Rhetoric
“Forgive if me I am getting Theranos vibes.”
Our reading: Marcus says he is getting Theranos vibes from OpenAI's handling of the o3 results.
No horizon
Qualifier: vibes
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“some possible explanations are given, but this is again a very bad look.”
Our reading: The gap between OpenAI's reported 32% and Epoch's 11% for o3-mini on FrontierMath is again a very bad look for OpenAI, even though some possible explanations have been given.
No horizon
Qualifier: some possible explanations are given
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“The truth is that we don’t really know how good o3 is or isn’t, and nobody should ever take OpenAI’s video presentations particularly seriously again, until they have been fully vetted by the community.”
Our reading: We don't really know how good o3 is or isn't, and nobody should ever again take OpenAI's video presentations particularly seriously until the community has fully vetted them.
No horizon
Qualifier: until they have been fully vetted by the community
Assessment: Can’t be checkedSaid
Rhetoric
“And given that Meta also appears to have just juiced some benchmarks, the whole thing is starting to look like a bunch of over-promisers scrambling to make things look better than they really are.”
Our reading: Given that Meta also appears to have just juiced some benchmarks, the whole thing is starting to look like a bunch of over-promisers scrambling to make things look better than they really are.
No horizon
Qualifier: Meta appears to have just juiced some benchmarks
Held from this post (6)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: The best stable ARC result the ARC team could get from the latest batch of publicly-testable OpenAI models was 56%, with a different model called o3-medium; this is still impressive and useful but a long way from the 75% OpenAI advertised.
- The criteria are ambiguous. Our reading, not assessed: o3 sometimes apparently cheats by reporting answers found online rather than computing them.
- The two reviews reached different assessments. Our reading, not assessed: None of OpenAI's released o3-named models can reliably do what was shown in the December livestream.