“AI and AI test sets have a huge English-language bias.”
Our reading, the sentence that was checked:
AI systems and AI test sets have a huge English-language bias.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, How Not to Test GPT-3, paragraph 52.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-how-not-to-test-gpt-3-d7578c-a1
We recorded 12 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-02-17. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- AI and AI test sets
- a huge English-language bias
- DO run tests in other languages
Our interpretation
- In context, AI refers to GPT-like language systems and their language and reasoning evaluations, not every kind of artificial intelligence.
- The plain reading is substantial English predominance in model development and evaluation coverage, with weaker representation of other languages. It does not assert that every model or benchmark is English-only.
- Huge describes a conspicuous imbalance; no numerical cutoff or population-weighting formula is supplied.
- For this ongoing research-state observation, use the latest completed research year and the elapsed publication year: 2022 through 17 February 2023. This observation window is evaluator-selected.
Counts as supported if: Primary documentation shows substantial English predominance in the training or tuning of prominent contemporary language models and in major general-purpose evaluation suites.
Counts as contradicted if: Representative documentation shows broadly balanced language coverage in both contemporary language-model development and major evaluation suites, without substantial English predominance.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Primary documentation shows a conspicuous English imbalance on both sides of the claim. InstructGPT's instruction data was overwhelmingly English, BIG-G's training mixture contained little non-English text, and HELM's initial core evaluations covered only English. BIG-bench also acknowledged limited lower-resource coverage. BLOOM and xP3 are important multilingual exceptions, so the supported reading is broad predominance rather than a universal English-only rule.
Limitations
- The assessment concerns language systems in the post's context, not all AI applications.
- Instruction tuning, pretraining, and evaluation coverage measure different aspects of the imbalance.
- No exact industry-wide English share or universal performance gap is established.
Evidence (4 items)
Supports the claim: Ouyang et al., OpenAI, arXiv,
almost entirely of English instructions
Why it matters here: The developer reports that roughly 96% of its instruction dataset was classified as English, possibly more because of classifier errors. This directly supports English predominance in instruction tuning, without treating that percentage as the pretraining distribution.
- Publisher
- Ouyang et al., OpenAI, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: Liang et al., Stanford CRFM, arXiv,
English varieties and dialects
Why it matters here: HELM's initial core scenarios covered only English, including different English varieties. The authors explicitly say that even multilingual BLOOM was evaluated only for English.
- Publisher
- Liang et al., Stanford CRFM, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: Srivastava et al., BIG-bench collaboration, arXiv,
a handful also target lower-resource languages
Why it matters here: BIG-bench documents limited lower-resource-language coverage and reports that its BIG-G training mixture contained about 6% non-English text. Its multilingual tasks also show that English predominance coexisted with deliberate efforts to test other languages.
- Publisher
- Srivastava et al., BIG-bench collaboration, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: BigScience collaboration, arXiv,
covering 46 languages
Why it matters here: BLOOM was trained on 46 natural languages, and the project developed a multilingual prompted dataset collection. These are substantial exceptions to an English-only description, but do not negate predominance in other prominent systems and evaluations.
- Publisher
- BigScience collaboration, arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, How Not to Test GPT-3
AI and AI test sets have a huge English-language bias.
- Where
- paragraph 52
- Published
- 2023-02-17
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-how-not-to-test-gpt-3-d7578c-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.