“Because of all that, we don’t know what explains its success, and we don’t know how to predict its failures:”
Our reading, the sentence that was checked:
Because GPT-4's size, architecture, energy use, processors, and training data have not been disclosed, we don't know what explains its success and we don't know how to predict its failures.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, GPT-4’s successes, and GPT-4’s failures, paragraph 13.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-gpt-4s-successes-and-gpt-4s-failures-c7bac3-a1
We recorded 12 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-03-15. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Causal claim
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- because of all that (the lack of disclosure) (qualifier, stated by the speaker)
From the speaker
- Because of all that
- we don’t know what explains its success
- we don’t know how to predict its failures
Our interpretation
- The preceding discussion identifies withheld technical information as the asserted cause.
- We means the outside research community. The plain reading is that missing information prevents a reliable account of GPT-4's successes and reliable advance identification of its failures.
- This does not require perfect prediction of every output, but identifying failures after they occur is insufficient.
- The relevant knowledge state is at launch and publication, March 14–15, 2023. OpenAI's private predictive abilities are relevant qualifications, not direct demonstrations of what outsiders could do.
Counts as supported if: Evidence identifies specific explanatory or predictive work blocked by the withheld information and establishes that missing access materially caused the stated inability.
Counts as contradicted if: Publicly reproducible work available by publication supplies reliable explanations and validated advance predictions of failures without the withheld information, or evidence shows the asserted inability did not result from nondisclosure.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Does the evidence support the cause?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The release materials supplied partial explanations and some successful aggregate predictions, while acknowledging remaining uncertainty. That qualifies a literal assertion that nothing could be explained or predicted. It does not establish what outsiders could reliably predict, or whether nondisclosure caused their limitations. Missing details alone do not prove the causal claim, and OpenAI's internal results do not decisively refute it.
Limitations
- The strongest quantitative evidence comes from OpenAI's own evaluations.
- Aggregate performance prediction is different from predicting particular failures.
- Later advances cannot establish what was publicly known when Marcus wrote.
Evidence (3 items)
Context only: OpenAI,
We developed infrastructure and optimization that have very predictable behavior across multiple scales.
Why it matters here: The launch account reports successful advance predictions of training loss and coding benchmark performance, alongside a partial explanation separating pretraining capabilities from later behavioral tuning. These company results qualify a blanket claim of ignorance but do not demonstrate independent prediction of individual failures.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: OpenAI,
Some capabilities are still hard to predict.
Why it matters here: OpenAI acknowledged remaining predictive uncertainty. It did not attribute that uncertainty to what it withheld from outside researchers.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: OpenAI,
The post-training hurts calibration significantly.
Why it matters here: The report compares model confidence with correctness before and after post-training, providing a limited explanation of unreliable confidence. This is evidence of partial understanding, not a test of whether disclosure would enable reliable external failure prediction.
- Where
- Figure 8 caption
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
What would settle it
Contemporaneous reproducible explanations and prospective failure predictions using only public information, or documented comparisons showing that access to the withheld details materially enabled work otherwise blocked.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, GPT-4’s successes, and GPT-4’s failures
Because of all that, we don’t know what explains its success, and we don’t know how to predict its failures:
- Where
- paragraph 13
- Published
- 2023-03-15
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-gpt-4s-successes-and-gpt-4s-failures-c7bac3-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-gpt-4s-successes-and-gpt-4s-failures-c7bac3-a1.