“It can do some things better, some worse, very few reliably.”
Our reading, the sentence that was checked:
Compared with a smart high schooler, GPT-4 can do some things better and some things worse, and it can do very few things reliably.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, AGI by 2027?, paragraph 5.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-agi-by-2027-341718-a2
We recorded 6 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-06-05. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- compared with a smart high schooler (for better/worse) (scope, stated by the speaker)
From the speaker
- some things better
- some worse
- very few reliably
- a smart high schooler
Our interpretation
- The ordinary reading is that GPT-4 has uneven abilities and can be depended on across only a small minority of relevant tasks.
- The comparison concerns capable high-school students, not average adults or professional specialists.
- I treat this as a snapshot immediately before publication. GPT-4 has no reporting quarter, so earlier evaluations are historical context rather than automatically decisive evidence of its June 2024 state.
- Reliability means consistent successful performance across varied instances of a task. It does not require absolute infallibility.
Counts as supported if: Representative evaluations of the relevant GPT-4 version and capable high-school students show both advantages and disadvantages, with dependable GPT-4 performance confined to a small minority of task categories.
Counts as contradicted if: Representative evaluations show dependable GPT-4 performance across a substantial breadth of relevant tasks, or otherwise disprove the asserted pattern of relative abilities.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
OpenAI’s March 2023 technical report reported human-level GPT-4 performance on the majority of the professional and academic exams it tested [e2]. Its March 14, 2023 disclosure acknowledged factual hallucinations and reasoning errors [e1]. These earlier findings establish reported strengths and limitations, but neither item measures GPT-4’s abilities against a matched group of capable high-school students or the proportion of tasks it performed reliably on June 5, 2024 [e1][e2]. They therefore leave the claim unresolved [e1][e2].
Limitations
- Both sources are company disclosures from 2023 and do not establish the performance of an identified June 2024 GPT-4 version [e1][e2].
- Exam scores do not establish repeated success across a representative range of relevant tasks or a matched comparison with capable high-school students [e2].
- Acknowledged factual and reasoning errors do not establish that only a small minority of task categories were reliable [e1].
Evidence (2 items)
Context only: OpenAI,
not fully reliable
Why it matters here: OpenAI acknowledged factual and reasoning failures. This concerns the original release and does not establish that only a small minority of tasks were reliable in June 2024.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
Context only: OpenAI,
human-level performance
Why it matters here: The report documents substantial exam performance, alongside weaker results on some exams. Exam scores neither establish consistent real-world success nor identify a representative population of smart high-school students.
- Publisher
- OpenAI
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- Describes events of
- Public from
What would settle it
Representative evaluations of an identified GPT-4 version available on June 5, 2024, using a matched group of capable high-school students, disclosed prompting and tools, and repeated trials across defined task categories. A specified reliability threshold and task denominator would allow assessment of both relative abilities and the breadth of dependable performance.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, AGI by 2027?
It can do some things better, some worse, very few reliably.
- Where
- paragraph 5
- Published
- 2024-06-05
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-agi-by-2027-341718-a2: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.