o3, AGI, the art of the demo, and what you can expect in 2025
We recorded 15 claims from this post. 7 were checked against evidence. 8 can’t be checked. 4 more are held. Read the post at its source.
Claims from this post (15)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
Observation
“(It certainly was not the long-awaited GPT-5).”
Our reading: OpenAI's o3 is not the long-awaited GPT-5.
No horizon Evidence as of
Limitation: The core evidence for the demo day is a news report, not an OpenAI page from that day.
Assessment: SupportedSaid
Observation
“The best results were truly impressive, such as major advances on Francois Chollet’s ARC task and on an important mathematical benchmark FrontierMath.”
Our reading: o3's best demo results showed truly impressive major advances on ARC and FrontierMath.
No horizon Evidence as of
Limitation: 'Truly impressive' is a judgment and was not scored.
Assessment: SupportedSaid
Observation
“The product was not yet released to the general public; it’s not been vetted by the scientific community.”
Our reading: As of the demo, o3 had not been released to the public nor vetted by the scientific community.
No horizon Evidence as of
Limitation: 'Vetted by the scientific community' has no precise test. ARC Prize's evaluation is a partial independent check, which he acknowledged.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Until lots of people get to try o3 on many different kinds of tasks, we should not assume that it is reliable.”
Our reading: People should not assume o3 is reliable until many people have tested it on many kinds of tasks.
No horizon
Qualifier: until many people test it on many kinds of tasks
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Spending a quarter trillion dollars in order to have something 1000 times more expensive than average humans is maybe not the way to go.”
Our reading: Spending a quarter trillion dollars to get systems 1000x more expensive than humans is maybe not the right approach.
No horizon
Assessment: Can’t be checkedSaid
Forecast
“Furthermore, the impact of all this on the environment could be immense.”
Our reading: The environmental impact of such high-compute AI systems could be immense.
No horizon
Assessment: Can’t be checkedSaid
Observation
“As in O3 may work for problems where massive data augmentation is possible, but not in others where it’s not.”
Our reading: o3 may work on problems allowing massive data augmentation but not on problems that do not.
No horizon
Assessment: UnresolvedSaid
Observation
“Most commentary has ignored one or both of these ideas.”
Our reading: Most commentary on o3 has ignored one or both of two ideas: that passing ARC may be necessary but not sufficient for AGI, and that a mechanism can be effective in some domains but not others.
No horizon Evidence as of
Limitation: The packet provides no representative sampling frame, denominator, systematic coding, or uncertainty estimate for commentary published between the December 20 announcement and Marcus’s December 21 post.
Assessment: UnresolvedSaid
Observation
“Even setting aside the 1000x costs, not one person outside of OpenAI has evaluated o3’s robustness across different types of problems.”
Our reading: No one outside OpenAI has evaluated o3's robustness across different problem types.
No horizon Evidence as of
Limitation: A broader reading of problem types could count ARC tasks and produce a different assessment.
Assessment: UnresolvedSaid
Forecast
“In 2025, a lot of AI influencers and maybe some companies are going to claim we have reached AGI.”
Our reading: During 2025, many AI influencers and possibly some companies will claim AGI has been reached.
Deadline passed Evidence as of
Limitation: The evidence is a selected set of statements, not a census.
Assessment: Can’t be checkedSaid
Forecast
“I wish that others would give lines in the sand in advance; instead, they are likely to craft definitions post hoc, or not give them at all.”
Our reading: Those claiming AGI are likely to craft definitions post hoc or give none at all.
No horizon
Assessment: Can’t be checkedSaid
Observation
“Nothing in the o3 presentation yesterday suggested to me that this decades-old core problem has been remotely solved.”
Our reading: Nothing in the o3 presentation suggested to Marcus that the decades-old core problem of neural networks generalizing better within distribution than outside it has been remotely solved.
No horizon Evidence as of
Limitation: This outcome makes no judgment about whether the presentation established a general solution to distribution shift.
Assessment: SupportedSaid
Observation
“I am saying that many basic problems haven’t been solved.”
Our reading: Many basic problems on the path to AGI remain unsolved.
No horizon Evidence as of
Limitation: The scope and meaning of 'many' are inferred and materially affect the assessment.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“And I am saying, as every roboticist on the planet knows, that we shouldn’t take demos seriously, until purported products are actually released and subjected to outside scrutiny.”
Our reading: AI demos should not be taken seriously until the products are released and subjected to outside scrutiny.
No horizon
Qualifier: until products are released and externally scrutinized
Assessment: Can’t be checkedSaid
Observation
“I couldn’t agree more. Yesterday’s demo, impressive as it was, doesn’t change that.”
Our reading: The o3 demo does not change the view that AGI requires true logical reasoning rather than statistical tricks.
No horizon
Held from this post (4)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: At best, o3 will likely prove reliable for some kinds of problems but not others.
- The record does not meet the current rules. Our reading, not assessed: Neural networks tend to generalize better within their training distribution than outside it.
- The two reviews reached different assessments. Our reading, not assessed: In 2025, almost nobody claiming AGI has been reached will provide their definition of AGI.