““Scaling” systems by making them larger has helped in some ways, but not others; we still really cannot guarantee that any given system will be honest, harmless, or helpful, rather than sycophantic, dishonest, toxic or biased.”
Our reading, the sentence that was checked:
Scaling has helped only partially; no system can yet be guaranteed honest, harmless, or helpful.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, What if Generative AI turned out to be a Dud?, paragraph 19.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-what-if-generative-ai-turned-out-81abbd-a2
We recorded 23 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2023-08-13. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Scaling
- making them larger
- helped in some ways, but not others
- cannot guarantee
- honest, harmless, or helpful
- sycophantic, dishonest, toxic or biased
Our interpretation
- The plain reading is that greater scale brought uneven gains and did not establish dependable honesty, safety, and helpfulness across intended uses.
- Guarantee means general behavioral assurance, not a legal warranty or a claim about every conceivable restricted program.
- The listed undesirable behaviors are examples; the claim does not require every model to exhibit every example.
- This is a historical observation about demonstrated results, not an estimate of the isolated causal effect of parameter count.
- With no common reporting cycle, the current-system observation window is calendar 2023 through publication.
Counts as supported if: Contemporary scaling evaluations show gains alongside persistent failures or regressions, and current-system evaluations document remaining honesty, safety, or helpfulness limitations.
Counts as contradicted if: Contemporary evidence establishes dependable assurance across the listed behaviors from scaling, with the purported remaining limitations resolved.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The June 15, 2023 scaling study found that performance on its Memo Trap task worsened with increasing scale in most tested model series, with some exceptions [e1]. The Llama 2 report available July 19 acknowledged hallucinations, potentially harmful or biased outputs, and excessive refusals after safety tuning [e2]. Anthropic's July 11 announcement reported gains in coding, math and reasoning, but did not establish that scale alone caused them [e3]. Together, these findings support uneven progress with remaining behavioral limitations, rather than dependable assurance of honesty, harmlessness and helpfulness [e1, e2, e3].
Limitations
- The scaling results concern selected tasks and model series; they do not show that scaling generally fails.
- The Claude 2 comparison does not isolate parameter count or other changes in training scale.
- The evidence does not cover every deployed model, establish every listed failure in every model, or rule out reliable restricted applications.
Evidence (3 items)
Supports the claim: Ian R. McKenzie and colleagues, arXiv,
monotonic inverse scaling
Why it matters here: The study documents tasks where increasing scale worsened performance despite broader gains in prediction and other tasks. It also reports exceptions and trend reversals, so it does not establish that scaling generally fails.
- Publisher
- Ian R. McKenzie and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: Hugo Touvron and colleagues, Meta, arXiv,
harmful, offensive, or biased content
Why it matters here: The model report acknowledges remaining harmful outputs after mitigation, and the same section describes factuality limitations. These disclosures are inconsistent with general behavioral assurance.
- Publisher
- Hugo Touvron and colleagues, Meta, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: Anthropic,
coding, math, and reasoning
Why it matters here: Anthropic reported practical capability gains and improved safety measurements. These qualify any suggestion of no progress, but the announcement does not isolate scaling as their cause or establish a guarantee.
- Where
- Opening paragraph
- Publisher
- Anthropic
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, What if Generative AI turned out to be a Dud?
“Scaling” systems by making them larger has helped in some ways, but not others; we still really cannot guarantee that any given system will be honest, harmless, or helpful, rather than sycophantic, dishonest, toxic or biased.
- Where
- paragraph 19
- Published
- 2023-08-13
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-what-if-generative-ai-turned-out-81abbd-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-what-if-generative-ai-turned-out-81abbd-a2.