“But here’s the thing, we can’t know in advance what model is big enough (if any) for any given problem.”
Our reading, the sentence that was checked:
We can't know in advance which model, if any, is big enough for any given problem.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Seven replies to the viral Apple reasoning paper – and why they fall short, paragraph 8.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-seven-replies-to-the-viral-apple-4722b8-a2
We recorded 17 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-06-12. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- in advance (qualifier, stated by the speaker)
- if any (qualifier, stated by the speaker)
From the speaker
- we can’t know in advance
- what model is big enough
- if any
- for any given problem
Our interpretation
- The contextual reading is that no generally dependable method determines a sufficient model size for an arbitrary specified problem before testing the candidate model on it.
- This does not mean that all advance performance estimates are impossible.
- Big enough means sufficient for reliable success, although the speaker supplies no success threshold.
- Use June 2025 for the ongoing state of knowledge surrounding the post; this inferred period has no regular reporting cadence.
Counts as supported if: A sufficiently comprehensive contemporary technical assessment establishes that available methods cannot reliably determine sufficient model size for arbitrary specified problems.
Counts as contradicted if: A publicly validated method available in the target period reliably determines sufficient model size before target-model testing across the claimed general scope.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
A February 2025 study identifies obstacles to predicting downstream accuracy from scale [e1]. An October 2024 version of another study reports successful preregistered predictions for subsequently released models on selected tasks [e2]. A June 2024 study predicts average downstream error using smaller experiments [e3]. These results establish neither that a generally dependable sizing method was unavailable in June 2025 nor that a validated method could determine sufficient size for arbitrary specified problems [e1, e2, e3].
Limitations
- Selected-task forecasts and average-error predictions do not establish reliable success on an arbitrary individual problem.
- The packet lacks a comprehensive assessment of methods available by June 2025.
- The claim leaves sufficient success and whether probabilistic forecasts count as knowing undefined.
Evidence (3 items)
Context only: arXiv; Schaeffer and colleagues,
progressively deteriorate
Why it matters here: The study identifies obstacles to predicting downstream accuracy from scale. It does not establish that all possible advance methods fail or that none existed by the post.
- Publisher
- arXiv; Schaeffer and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Ruan, Maddison, and Hashimoto,
preregistered predictions
Why it matters here: Successful prospective predictions challenge a reading that denies all advance knowledge. They predict selected task performance using capability measurements, rather than guaranteeing a sufficient parameter count for an arbitrary individual problem.
- Publisher
- arXiv; Ruan, Maddison, and Hashimoto
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Gadre and colleagues,
downstream task performance
Why it matters here: The study predicts average downstream error using smaller experiments. This is useful advance information, but an average error prediction does not certify success on any specified problem.
- Where
- Abstract
- Publisher
- arXiv; Gadre and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
A comprehensive technical assessment of methods available by June 2025 establishing the general limitation, or a method validated by then that determines sufficient model size before target-model testing across arbitrary specified problems, with explicit success and uncertainty criteria.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Seven replies to the viral Apple reasoning paper – and why they fall short
But here’s the thing, we can’t know in advance what model is big enough (if any) for any given problem.
- Where
- paragraph 8
- Published
- 2025-06-12
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-seven-replies-to-the-viral-apple-4722b8-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.