“In fact, if you simply collapsed all the measurements of accuracy, and did the single most obvious test here, a simple t-test, the results would (as Footnote C implies) be significant.”
Our reading, the sentence that was checked:
If all the accuracy measurements in OpenAI's study were collapsed and tested with a simple t-test, the results would be statistically significant, as Footnote C implies.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Diving deep into OpenAI’s new study on LLM’s and bioweapons, paragraph 15.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-when-looked-at-carefully-openais-483f6c-a2
We recorded 22 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-02-04. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- if all accuracy measurements were collapsed (qualifier, stated by the speaker)
- using a simple t-test (qualifier, stated by the speaker)
- as Footnote C implies (qualifier, stated by the speaker)
From the speaker
- collapsed all the measurements of accuracy
- a simple t-test
- would ... be significant
- as Footnote C implies
Our interpretation
- This is a testable statement about an existing dataset, not a conditional forecast.
- Collapse means sum the five accuracy scores within each participant, preserving participants as independent observations.
- The primary test combines both cohorts and compares the 50 LLM-assigned participants with the 50 internet-only participants.
- Use an unadjusted, two-sided Welch t-test with significance at p < 0.05; also check the expert-only interpretation suggested by the preceding discussion.
- This assesses the numerical significance claim, not whether this was the uniquely correct analysis.
Counts as supported if: A participant-level t-test of total accuracy produces p < 0.05 for the combined sample, with a positive LLM-group difference.
Counts as contradicted if: The specified participant-level t-test produces p >= 0.05.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
A reanalysis of OpenAI's data released January 31, 2024 sums five accuracy scores per participant and combines the expert and student cohorts [e1][e2]. Mean totals were 22.98 with model access and 19.94 with internet alone, with 50 participants per group [e1][e2]. The unadjusted, two-sided Welch t-test gives p = 0.01908, supporting the aggregate significance claim at p < 0.05 [e1]. OpenAI's published summary separately reports a one-sided expert-total p-value of 0.0027 [e3].
Limitations
- The result does not establish that this analysis was uniquely appropriate or that multiple-comparison adjustment was unnecessary.
- The published student-total one-sided p-value is 0.2399; significance is not established for both cohorts separately [e3].
- The supplied evidence items do not reproduce Footnote C, so its wording is not independently verified here.
Evidence (3 items)
Supports the claim: OpenAI,
id,stage,accuracy,completeness,innovation,difficulty,n_user_messages,n_internet_resources,time_to_complete_sec 0,ideation,7,2,1,6,0,23,2075
Why it matters here: Summing within participants and comparing the two assignment groups gives a positive difference with two-sided p = 0.01908. This remains significant while respecting repeated measurements.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Arithmetic
- Sum five accuracy scores per participant. Welch t = (22.98 - 19.94) / sqrt(6.523083032194053^2 / 50 + 6.228342066550229^2 / 50) = 2.383410643; Welch-Satterthwaite df = 97.791234; two-sided Student-t p = 0.01908457. (inputs: All 500 accuracy records from responses.csv; All 100 assignment records from participants.csv, joined by id; LLM group: n = 50, mean total = 22.98, sample SD = 6.523083032194053; Internet-only group: n = 50, mean total = 19.94, sample SD = 6.228342066550229; Displayed p rounded to five decimal places; calculations used unrounded participant totals.)
- Describes events of
- Public from
Context only: OpenAI,
id,cohort,assignment,llm_experience 0,expert,llm_internet,Used a few times 1,expert,internet_only,Use at least once every few weeks
Why it matters here: This supplies the cohort and assignment labels needed for e1's calculation.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: OpenAI,
,T-test p-value,0.3998,0.9584,0.0445,0.0683,0.3170,0.2399,0.2165,0.0721,0.0073,0.0160,0.1042,0.0027
Why it matters here: The published one-sided expert total test has p = 0.0027, whereas the student total has p = 0.2399. The expert-only reading also supports significance, but a claim of significance within both cohorts separately would not hold.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Diving deep into OpenAI’s new study on LLM’s and bioweapons
In fact, if you simply collapsed all the measurements of accuracy, and did the single most obvious test here, a simple t-test, the results would (as Footnote C implies) be significant.
- Where
- paragraph 15
- Published
- 2024-02-04
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-when-looked-at-carefully-openais-483f6c-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-when-looked-at-carefully-openais-483f6c-a2.