“More data won’t solve that problem.”
Our reading, the sentence that was checked:
More training data will not solve Sora's hallucination-like physics glitches.
Assessment: Unresolved No deadline stated on 2026-10-29 Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Sora’s Surreal Physics, paragraph 22.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-soras-surreal-physics-cb9e51-a2
We recorded 14 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-02-16. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Forecast
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No deadline stated
- Deadline state on 2026-10-29
- No deadline stated (the release’s reference date)
From the speaker
- More data
- won’t solve
- that problem
Our interpretation
- That problem refers to recurring physics and object-permanence glitches described immediately beforehand.
- More data means increasing training examples without introducing a new architecture or explicit physical constraints.
- Solve means reliably resolving these failure modes across varied situations, rather than improving selected examples.
- No deadline or numerical success threshold is supplied.
Counts as supported if: A valid general result establishes that data-only scaling of the relevant Sora approach cannot resolve these failure modes.
Counts as contradicted if: A controlled data-only scaling experiment on the relevant Sora approach reliably resolves the described failure modes across varied, previously unseen situations.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Did the forecast come true?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No deadline stated
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The November 4, 2024 preprint reports generalization successes and out-of-distribution failures in research models; these finite experiments do not establish impossibility for Sora [e1]. The March 12, 2025 preprint reports improved falling-object behavior from data-only fine-tuning of Open-Sora, but does not demonstrate a general solution for OpenAI's Sora [e2]. OpenAI's September 30, 2025 announcement reports improved physics in Sora 2, without isolating the effect of additional data or establishing that the specified failures are solved [e3]. These results provide neither a qualifying data-only solution nor an applicable general impossibility result, so they do not settle the forecast [e1, e2, e3].
Limitations
- The research experiments test models other than OpenAI's Sora and do not establish what all larger datasets could achieve [e1, e2].
- Improved falling-object behavior does not establish reliable physical consistency and object persistence across varied situations [e2].
- The Sora 2 announcement does not isolate data-only scaling or provide independent tests establishing resolution of the specified failures [e3].
Evidence (3 items)
Context only: arXiv; Bingyi Kang and colleagues,
failure in out-of-distribution scenarios.
Why it matters here: The experiments identify limits to physics generalization alongside successful generalization in other settings. They test research models rather than OpenAI's Sora and do not prove that every larger training dataset must fail.
- Where
- Abstract
- Publisher
- arXiv; Bingyi Kang and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Chenyu Li and colleagues,
realistic dropping behavior
Why it matters here: Additional simulated training examples improve falling-object behavior without changing Open-Sora's training objective. This demonstrates that data can remedy particular physical behaviors, but the model and task do not match a general solution for OpenAI's Sora.
- Publisher
- arXiv; Chenyu Li and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: OpenAI,
more physically accurate
Why it matters here: OpenAI reports improved physics in a later model, while acknowledging remaining mistakes. The announcement does not attribute the improvement to data alone or demonstrate that the relevant glitches have been solved.
- Where
- Opening description
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
What would settle it
A controlled experiment on the relevant Sora approach showing that additional data alone reliably resolves the specified failures across varied, previously unseen situations, documenting model versions, data quantities, other training changes, and independent tests; or a valid general impossibility result applicable to that approach.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Sora’s Surreal Physics
More data won’t solve that problem.
- Where
- paragraph 22
- Published
- 2024-02-16
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-soras-surreal-physics-cb9e51-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.