“Machines often solve them for the wrong reasons. Building them is an unsolved problem within AI.”
Our reading, the sentence that was checked:
Machines often solve benchmarks for the wrong reasons and building good benchmarks is an unsolved problem.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, No, Virginia, AGI is not imminent, paragraph 14.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-no-virginia-agi-is-not-imminent-6ab6b8-a1
We recorded 20 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-08-20. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Machines often solve them for the wrong reasons.
- Building them is an unsolved problem within AI.
Our interpretation
- The antecedent of 'them' is AI benchmarks.
- The ordinary reading is that shortcut solutions recur and designing benchmarks that reliably measure the intended abilities remains an open research problem. This does not mean that no useful benchmarks exist.
- 'Often' means a recurring, documented problem across tasks, not a specified percentage of all benchmark results.
- The statement concerns the research state at publication. With no regular reporting period for this subject, the target period is the current calendar year through August 20, 2023.
Counts as supported if: Research describing the target period documents recurring shortcut solutions and continuing difficulty constructing benchmarks that measure their intended abilities.
Counts as contradicted if: Broad validation shows that shortcut solutions are exceptional and that reliable benchmark construction is generally a resolved problem.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Contemporaneous research documents persistent benchmark shortcuts and continuing difficulty removing them. The NLI audit and prompt-learning experiments establish recurrence across tasks. Useful robustness benchmarks also existed, so the evidence supports the ordinary reading that reliable benchmark design remained an open research problem, not that every benchmark was worthless.
Limitations
- No numerical prevalence is established for 'often'.
- The evidence supports the contextual language-model claim more directly than every area of AI.
- The PECO work began before 2023; its dated 2023 version is used for its continuing assessment of benchmark design.
Evidence (3 items)
Supports the claim: arXiv; Michael Saxon and colleagues,
free from shortcut biases
Why it matters here: The 2023 version describes benchmark design as an ongoing difficulty and reports that single-sentence label leakage persists despite mitigation efforts. The event date denotes that research-state assessment, not the date of every underlying experiment.
- Publisher
- arXiv; Michael Saxon and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: arXiv; Ruixiang Tang and colleagues,
shortcuts in prompts
Why it matters here: Experiments across classification and extraction tasks found reliance on spurious prompt cues. Together with the NLI audit, this supports recurrence across tasks rather than an isolated defective benchmark.
- Publisher
- arXiv; Ruixiang Tang and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Mengnan Du and colleagues,
benchmark datasets for adversarial and OOD robustness
Why it matters here: The review acknowledges useful robustness benchmarks while calling for broader task and bias coverage. This limits an absolute reading of 'unsolved' but is consistent with benchmark construction remaining an active research problem.
- Publisher
- arXiv; Mengnan Du and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, No, Virginia, AGI is not imminent
Machines often solve them for the wrong reasons. Building them is an unsolved problem within AI.
- Where
- paragraph 14
- Published
- 2023-08-20
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-no-virginia-agi-is-not-imminent-6ab6b8-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-no-virginia-agi-is-not-imminent-6ab6b8-a1.