“Even with a lot of data, GPT-4 is playing chess at around the level where (top) chess programs were in 1976, not even close to current state of the art.”
Our reading, the sentence that was checked:
GPT-4 plays chess at about the level of top chess programs in 1976, far from the current state of the art.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Could GPT-5 revolutionize military strategy?, paragraph 8.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-could-gpt-5-revolutionize-military-846b37-a2
We recorded 8 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-02-07. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- even with a lot of data (qualifier, stated by the speaker)
From the speaker
- Even with a lot of data
- GPT-4
- playing chess
- around the level where (top) chess programs were in 1976
- not even close to current state of the art
Our interpretation
- The ordinary reading concerns GPT-4 playing complete chess games without an external chess engine.
- Both comparisons matter: approximate equivalence to leading 1976 programs and a substantial deficit against leading contemporary engines.
- The target is the capability snapshot at publication, the latest state that pre-publication evidence could describe. There is no standard reporting period for this capability. Earlier experiments are historical context unless their applicability to this snapshot is established.
- Around permits ordinary measurement uncertainty; it does not supply a numerical Elo tolerance.
- Even with a lot of data describes existing training exposure, not a conditional forecast or an instruction to supply unlimited assistance.
Counts as supported if: Comparable, representative measurements establish that the relevant GPT-4 version played roughly as strongly as leading 1976 programs and substantially below leading engines at publication.
Counts as contradicted if: Comparable measurements establish materially different strength from leading 1976 programs, or competitive strength against leading engines at publication.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
A September 2023 evaluation estimated GPT-4 at 1305 Elo counting all games and 1371 counting only legal games, with cautions about engine calibration [e1]. A historical account reports a 1950 tournament performance rating for CHESS 4.5 in 1976 [e2]. These figures lack a validated common scale, so they establish neither approximate equivalence nor a material difference [e1, e2]. The earlier GPT-4 tests also do not establish its strength on February 7, 2024, against either leading 1976 programs or leading contemporary engines [e1, e2].
Limitations
- The 1976 tournament figure comes from a secondary account [e2].
- The September 2023 evaluation is not verified as representative of the publication-date GPT-4 configuration [e1].
- The pretraining paper confirms exposure to chess games but does not quantify that exposure or establish complete-game playing strength [e3].
Evidence (3 items)
Context only: Mathieu Acher,
30 were illegal games
Why it matters here: The evaluation estimates GPT-4 at 1305 Elo when all games count and 1371 when only legal games count; it assigns the higher roughly 1750 estimate to GPT-3.5-turbo-instruct. These earlier results document weakness, but the author warns about Stockfish calibration and they do not establish a common scale with 1976 tournament ratings.
- Publisher
- Mathieu Acher
- Kind of source
- Independent measurement
- Describes events of
- Public from
Context only: Chess.com, Bill Wall,
The performance rating
Why it matters here: This supplies a historical tournament-performance estimate, not a rating directly interchangeable with Acher's engine-calibrated estimate. It is secondary context because the original records sought could not be read.
- Publisher
- Chess.com, Bill Wall
- Kind of source
- Secondary summary
- Describes events of
- Public from
Context only: arXiv; Collin Burns and colleagues,
included chess games
Why it matters here: The primary paper confirms chess training exposure. It does not quantify that exposure here or establish GPT-4's complete-game strength.
- Publisher
- arXiv; Collin Burns and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Version-identified, complete-game GPT-4 results applicable to February 7, 2024, compared with leading 1976 programs and contemporary engines under matched conditions or a validated common rating scale, with prompting, opponent settings, time controls, and illegal-move rules documented.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Could GPT-5 revolutionize military strategy?
Even with a lot of data, GPT-4 is playing chess at around the level where (top) chess programs were in 1976, not even close to current state of the art.
- Where
- paragraph 8
- Published
- 2024-02-07
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-could-gpt-5-revolutionize-military-846b37-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-could-gpt-5-revolutionize-military-846b37-a2.