“Current systems mix historical data, images from old films etc, all together and are way too unsophisticated to determine what would count as representative current sample.”
Our reading, the sentence that was checked:
Current AI systems mix historical and current data indiscriminately and are too unsophisticated to determine a representative current sample.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Has Google gone too woke? Why even the biggest models still struggle with guardrails, paragraph 18.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-has-google-gone-too-woke-why-even-6d5f86-a1
We recorded 12 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-02-21. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Current systems
- mix historical data, images from old films etc, all together
- way too unsophisticated
- representative current sample
Our interpretation
- The relevant systems are contemporary image generators discussed in the post.
- I read representative current sample as generated population distributions reasonably matching an identified contemporary population, rather than every small image batch matching exact proportions.
- The claim combines mixed-era training data with unreliable contemporary representation; evidence of biased outputs alone does not establish both.
- Because no reporting interval is supplied, the target is the current deployment period in February 2024 through publication on February 21.
Counts as supported if: Training documentation establishes mixed historical and contemporary inputs, and audits of the relevant deployments show persistent failure to represent specified contemporary populations.
Counts as contradicted if: Documentation disproves the asserted data mixing, or representative audits show that the relevant systems reliably produce contemporary population samples when the intended population is specified.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The sources establish reasons for concern about demographic representation, but do not settle the complete claim for February 2024. The earlier surgeon audit found substantial failures alongside a successful match to attending-surgeon demographics. Dataset documentation does not establish how the relevant systems handled historical material or selected contemporary populations.
Limitations
- The demographic experiments predate the target period.
- The dataset documentation is not a complete account of the relevant commercial systems.
- The audit's January 2024 issue date is used as a conservative availability date.
Evidence (2 items)
Context only: LAION,
this large-scale dataset is uncurated.
Why it matters here: LAION documents a broadly collected web dataset. This does not by itself establish the mixture of depicted eras, use of old films, or contemporary sampling ability of the systems Marcus meant.
- Publisher
- LAION
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: JAMA Surgery, abstract hosted by PubMed,
underestimated trainee representation.
Why it matters here: The May 2023 study found demographic mismatches but also a match between DALL-E 2 outputs and attending surgeons. The result depends on the reference population and does not directly measure February 2024 behavior.
- Publisher
- JAMA Surgery, abstract hosted by PubMed
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
What would settle it
Version-matched training documentation and February 2024 demographic audits with explicit contemporary populations and adequate repeated sampling.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Has Google gone too woke? Why even the biggest models still struggle with guardrails
Current systems mix historical data, images from old films etc, all together and are way too unsophisticated to determine what would count as representative current sample.
- Where
- paragraph 18
- Published
- 2024-02-21
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-has-google-gone-too-woke-why-even-6d5f86-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-has-google-gone-too-woke-why-even-6d5f86-a1.