โBecause the wrong experiment was performed, and key data werenโt given, we canโt compare directly with humans. (And best humans still outperformed the model).โ
Our reading, the sentence that was checked:
Because the wrong experiment was performed and key data were not given, o3 cannot be compared directly with humans on ARC, and the best humans still outperformed the model.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-c39-838202-a2
We recorded 17 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-12-22. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the releaseโs reference date)
From the speaker
- the wrong experiment was performed
- key data werenโt given
- we canโt compare directly with humans
- best humans still outperformed the model
Our interpretation
- Wrong experiment is read as a missing control relevant to the scientific comparison, not as a violation of benchmark rules.
- Direct comparison means a meaningful comparison of learning and generalization under sufficiently comparable protocols, not merely placing two percentages beside each other.
- Best humans ordinarily means strong individual human solvers. Pooling successful answers from different people is a distinct measure.
- Both the causal explanation and the asserted human advantage must be established to support the whole claim.
- The target is the December demonstration and the evidence available when Marcus discussed it.
Counts as supported if: Protocols establish that the missing ARC-untrained control prevents the claimed comparison of generalization, and matched results establish that top individual humans exceeded the demonstrated o3 system.
Counts as contradicted if: A suitable controlled comparison was available despite the alleged missing data, or matched results establish that o3 equaled or exceeded the relevant top human performance.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The study available on 2 September 2024 reports that 98.8% of tasks in each set were solved by at least one participant; this is pooled solvability, not a top individual's score [e1]. ARC Prize's 12 June 2024 statement that humans can solve all tasks rests on human verification of the tasks [e2]. A 20 December 2024 live blog reports a contrary launch statement that the model beat the top human result, but supplies no underlying human measurements [e3]. These sources establish neither the ranking of top individual humans against December o3 under comparable conditions nor whether the missing ARC-untrained control prevents the intended comparison of generalization [e1โe3].
Limitations
- No matched evaluation of December o3 and top individual human solvers is supplied.
- The contrary launch statement is a secondary report without a verified primary transcript or underlying human protocol.
- The packet does not isolate the effect of the missing training control from other possible protocol differences.
Evidence (3 items)
Context only: Solim LeGris and colleagues, arXiv,
solved by at least one person
Why it matters here: This is pooled task solvability across participants. It does not establish that a top individual exceeded o3 under matched conditions.
- Publisher
- Solim LeGris and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: ARC Prize,
humans can solve 100%
Why it matters here: The surrounding explanation bases this on human verification of tasks. It supports human solvability, but supplies no matched score for the best individual solvers in December.
- Publisher
- ARC Prize
- Kind of source
- Benchmark result
- Describes events of
- Public from
Context only: Simon Willison,
the top result for a human
Why it matters here: This contemporaneous account reports a statement contrary to Marcus's human-ranking assertion. Without the underlying human protocol or a verified transcript, it is material context rather than decisive measurement.
- Publisher
- Simon Willison
- Kind of source
- Secondary summary
- Describes events of
- Public from
What would settle it
A matched evaluation of the December o3 system and top individual humans, disclosing task splits, scoring, attempts, feedback and prior ARC exposure, together with controlled model results establishing whether the missing ARC-untrained control prevents the intended comparison of generalization.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, ๐ผ๐ฏ โ๐๐ฅ๐ ๐๐๐โ ๐ฝ๐ผ๐๐๐บ๐ผ๐ฟ๐๐ฒ๐บ ๐บ๐ฒ๐ด๐ฎ๐๐ต๐ฟ๐ฒ๐ฎ๐ฑ: ๐๐ต๐ ๐๐ต๐ถ๐ป๐ด๐ ๐ด๐ผ๐ ๐ต๐ฒ๐ฎ๐๐ฒ๐ฑ, ๐๐ต๐ฎ๐ ๐๐ฒ๐ป๐ ๐๐ฟ๐ผ๐ป๐ด, ๐ฎ๐ป๐ฑ ๐๐ต๐ฎ๐ ๐ถ๐ ๐ฎ๐น๐น ๐บ๐ฒ๐ฎ๐ป๐
Because the wrong experiment was performed, and key data werenโt given, we canโt compare directly with humans. (And best humans still outperformed the model).
- Where
- paragraph 14
- Published
- 2024-12-22
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-c39-838202-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.