๐ผ๐ฏ โ๐๐ฅ๐ ๐๐๐โ ๐ฝ๐ผ๐๐๐บ๐ผ๐ฟ๐๐ฒ๐บ ๐บ๐ฒ๐ด๐ฎ๐๐ต๐ฟ๐ฒ๐ฎ๐ฑ: ๐๐ต๐ ๐๐ต๐ถ๐ป๐ด๐ ๐ด๐ผ๐ ๐ต๐ฒ๐ฎ๐๐ฒ๐ฑ, ๐๐ต๐ฎ๐ ๐๐ฒ๐ป๐ ๐๐ฟ๐ผ๐ป๐ด, ๐ฎ๐ป๐ฑ ๐๐ต๐ฎ๐ ๐ถ๐ ๐ฎ๐น๐น ๐บ๐ฒ๐ฎ๐ป๐
We recorded 17 claims from this post. 10 were checked against evidence. 7 canโt be checked. 3 more are held. Read the post at its source.
Claims from this post (17)
In the order they appear in the post. Each quotation is the speakerโs words in this post. A label such as โSupportedโ or โCanโt be checkedโ is the claimโs current assessment; its page has the evidence and the reasons.
Assessment: SupportedSaid
Observation
โAt *most* the test is necessary for AGI; it certainly isnโt sufficient.โ
Our reading: Passing ARC is at most necessary for AGI and certainly not sufficient.
No horizon Evidence as of
Limitation: The contract notes that AGI is not defined, so 'sufficient' is judged by the designers' stated view.
Assessment: MixedSaid
Observation
โCritical things like factuality, compositionality, and common sense arenโt even addressed.โ
Our reading: The ARC test does not address factuality, compositionality, or common sense.
No horizon Evidence as of
Limitation: The result depends on how 'common sense' and 'compositionality' are defined. A narrower reading, such as broad everyday world knowledge, would favor Marcus.
Assessment: Canโt be checkedSaid
Recommendation or value judgment
โAs NYU prof Brenden Lake pointed out, the test should never have been called ARC-AGI.โ
Our reading: The ARC test should never have been named ARC-AGI.
No horizon
Assessment: Canโt be checkedSaid
Recommendation or value judgment
โThe video should have been much clearer about what was actually tested and what was actually trained.โ
Our reading: OpenAI's o3 announcement video should have been much clearer about what was tested and what was trained.
No horizon
Assessment: UnresolvedSaid
Observation
โSuch pretraining is not uncommon in the field, but was not made clear in the video.โ
Our reading: Such pretraining is not uncommon in the field, but OpenAI's video did not make clear that o3 had been pretrained on ARC examples.
No horizon Evidence as of
Limitation: Clarity requires considering slides as well as spoken words.
Assessment: SupportedSaid
Observation
โWhat was actually done - pretraining on what I believe was hundreds of public examples - is NOT comparable to what humans require.โ
Our reading: What was done for o3 on ARC, pretraining on what Marcus believes was hundreds of public examples, is not comparable to what humans require.
No horizon Evidence as of
Limitation: The conclusion depends on reading comparable as referring to ARC-specific preparation.
Assessment: Canโt be checkedSaid
Recommendation or value judgment
โBecause of the pretraining, and lack of comparability, what was actually shown was disappointing.โ
Our reading: Because of the pretraining and the lack of comparability with humans, what o3 actually showed on ARC was disappointing.
No horizon
Qualifier: because of the pretraining and lack of comparability
Assessment: UnresolvedSaid
Causal claim
โSame was true of the openAI graph: the MIT work (halfway in between o1 and o3) and many others results werenโt shown, making the breakthrough relative to the field seem far bigger than it really was.โ
Our reading: OpenAI's graph left out the MIT work (halfway between o1 and o3) and many other results, which made o3's breakthrough relative to the field seem far bigger than it really was.
No horizon Evidence as of
Limitation: The identified MIT figure is an ensemble result, not simply the standalone test-time-training model.
Assessment: UnresolvedSaid
Observation
โTwo graphs, one presented by OpenAI and one by Chollet were misleading.โ
Our reading: The o3 ARC graphs presented by OpenAI and by Chollet were misleading.
No horizon Evidence as of
Limitation: An image's absence from extracted text cannot establish an omission within that image.
Assessment: Canโt be checkedSaid
Recommendation or value judgment
โAs the scientist Adan Becerra, PhD put it (and Chollet publicly agreed) the best thing would have been to present data for the โbase modelโ without the pretraining.โ
Our reading: OpenAI should have presented ARC results for the o3 base model without the pretraining.
No horizon
Assessment: UnresolvedSaid
Observation
โUnfortunately the key test was not done.โ
Our reading: The key test of o3's base model on ARC without pretraining was not done.
No horizon Evidence as of
Limitation: A narrower reading confined to ARC Prize's own evaluation would be easier to support.
Assessment: SupportedSaid
Observation
โWe still donโt have a solid test of what o3 does without the pretraining, in the case that would be more comparable to humansโ
Our reading: There is no solid test yet of o3's ARC performance without pretraining.
No horizon Evidence as of
Limitation: The evaluator page has later updates, and the training note's precise insertion date is not separately archived in this record.
Assessment: UnresolvedSaid
Causal claim
โBecause the wrong experiment was performed, and key data werenโt given, we canโt compare directly with humans. (And best humans still outperformed the model).โ
Our reading: Because the wrong experiment was performed and key data were not given, o3 cannot be compared directly with humans on ARC, and the best humans still outperformed the model.
No horizon Evidence as of
Limitation: No matched evaluation of December o3 and top individual human solvers is supplied.
Assessment: Canโt be checkedSaid
Conditional forecast
โUntil there is considerable external scientific scrutiny (so far there has been none), we wonโt really know exactly what the o3 advance is or how important it is.โ
Our reading: Until o3 receives considerable external scientific scrutiny (so far it has had none), we won't really know exactly what the o3 advance is or how important it is.
No horizon
Only if: until there is considerable external scientific scrutiny
Assessment: UnresolvedSaid
Observation
โWhat we saw is not AGI.โ
Our reading: o3 as demonstrated is not AGI.
No horizon Evidence as of
Limitation: This assesses the categorical wording, not the narrower claim that ARC performance alone did not prove AGI.
Assessment: Canโt be checkedSaid
Recommendation or value judgment
โThe media should be asking hard questions, not fanning hype.โ
Our reading: The media should ask hard questions about AI instead of fanning hype.
No horizon
Assessment: Canโt be checkedSaid
Recommendation or value judgment
โPeople in the media probably shouldnโt even joke about o3 being AGI.โ
Our reading: People in the media probably shouldn't even joke about o3 being AGI.
No horizon Evidence as of
Limitation: Only this normative sentence is assessed. Nearby factual claims in the post are not covered.
Held from this post (3)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: o3 did not take the ARC test cold with only a few sample items as a human would.