“From the initial report, we can tell almost nothing.”
Our reading, the sentence that was checked:
From OpenAI's initial report, we can tell almost nothing about the new system.
Assessment: Mixed Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Brief remark on OpenAI’s giant math drop, paragraph 19.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-complementary-remarks-from-gary-marcus-4fd0e9-a1
We recorded 5 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2026-10-07. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- from the initial report (scope, stated by the speaker)
- almost nothing (qualifier, stated by the speaker)
From the speaker
- From the initial report
- we can tell almost nothing
- We don’t know what the procedure was
- We know nothing about the architecture
- We know nothing about the failure rate
- We know nothing about the training/post training/data augmention
- how generalizable the result is outside math
Our interpretation
- The plain reading concerns scientifically useful information about the system's operation, reliability, training, and generality, rather than the existence of mathematical manuscripts.
- The initial report includes the October 6 announcement and the repository it expressly directs readers to. Later additions are excluded.
- Assess the named dimensions separately. A mixed result is appropriate if important methodological gaps coexist with substantive information about the system's evaluation and operation.
- Attempt counts are informative but do not by themselves establish a failure rate.
Counts as supported if: The initial materials leave the named scientific questions substantially unanswered and provide negligible usable information for assessing how the system works or how broadly its performance generalizes.
Counts as contradicted if: The initial materials provide substantive information about the system's process, evaluation, reliability, training, or generality, making 'almost nothing' an overstatement for those dimensions.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Mixed
Part of the claim holds and part does not, and the record says which part is which.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Marcus's concern about incomplete scientific disclosure holds in part: prompts were missing and the model was unavailable. The inspected launch documents also leave major questions about training, architecture, failure rates, and generality unanswered. But 'almost nothing' overstates the lack of information about the evaluation: the release disclosed approximate compute, the number of problems attempted, and reasoning summaries. This is a mixed assessment of disclosure, not a judgment that the mathematical results were correct or that the system generalized beyond mathematics.
Limitations
- The outcome depends on reading the sentence broadly in light of the preceding methodological criticisms. A reading confined to generality beyond mathematics would differ.
- Company-reported evaluation details are disclosures, not independent verification of performance.
- The missing-prompts finding is corroborated by contemporaneous news reporting.
- Individual reasoning-summary files were not examined.
Evidence (3 items)
Supports the claim: Decrypt,
no prompts
Why it matters here: This contemporaneous report corroborates an important methodological omission: the prompts were not disclosed and the model was unavailable. It supports Marcus's concern about limited scrutiny, while also acknowledging information that was disclosed.
- Publisher
- Decrypt
- Kind of source
- News report
- Describes events of
- Public from
Goes against the claim: OpenAI,
approximately 4,000 problems.
Why it matters here: The initial README supplied an evaluation denominator and described selection into result families. This is useful evaluation information, although it does not establish a failure rate because attempts, families, and manuscripts are different units.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Goes against the claim: OpenAI,
roughly three hours
Why it matters here: The announcement disclosed an approximate compute budget and identified ten reasoning summaries and attempt statistics in the repository. These provide substantive, though incomplete, information about the evaluation.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Brief remark on OpenAI’s giant math drop
From the initial report, we can tell almost nothing.
- Where
- paragraph 19
- Published
- 2026-10-07
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-complementary-remarks-from-gary-marcus-4fd0e9-a1: Current. Outcome: Mixed. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-complementary-remarks-from-gary-marcus-4fd0e9-a1.