“Thus far we have seen consistently unreliable results, from math to fact-checking.”
Our reading, the sentence that was checked:
So far, the authors have seen consistently unreliable results from LLMs using external tools, from math to fact-checking.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Getting GPT to work with external tools is harder than you think, paragraph 22.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-getting-gpt-to-work-with-external-44d1f1-a1
We recorded 14 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2023-10-23. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- what the authors have seen thus far (scope, stated by the speaker)
From the speaker
- Thus far
- we have seen
- consistently unreliable results
- from math to fact-checking
Our interpretation
- The claim concerns recurrent failures in the authors' observations of tool-assisted mathematics and fact-checking through publication.
- Consistently unreliable means recurring errors, not that every answer was wrong; the surrounding text explicitly acknowledges successes.
- The observation period ends on publication. Its beginning is unspecified, so no lower bound is imposed.
- Independent tests on different questions can corroborate or qualify the general pattern but cannot establish what the authors themselves observed.
Counts as supported if: Independent records or replication of the observations underlying the post verify recurring errors in both tool-assisted mathematics and fact-checking.
Counts as contradicted if: Independent examination of those observations shows that the alleged recurring errors were absent, incorrectly scored, or attributed to the wrong systems.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Independent studies show both recurring tool-assisted errors and substantial successes. They make the general concern plausible, but they do not independently verify the authors' combined account of what they observed in mathematics and fact-checking. Scoring that account as supported would substitute different experiments for the claim's stated scope.
Limitations
- The outcome depends on retaining the explicit personal scope, 'we have seen'.
- Citation support is not equivalent to factual accuracy.
- Failure to locate independent verification does not show that the reported observations were false.
Evidence (2 items)
Context only: arXiv; Nelson F. Liu, Tianyi Zhang, and Percy Liang,
precision (89.5)
Why it matters here: Responses collected in February–March show imperfect citation support, including for Bing, which corroborates a general reliability concern. This measures citation support rather than factual truth, excludes Bard, and does not examine the authors' particular observations.
- Publisher
- arXiv; Nelson F. Liu, Tianyi Zhang, and Percy Liang
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Tanuj Kumar and Mikhail A. Kats,
correct on every problem
Why it matters here: An independent 13-problem study found that repeated runs and consensus selection answered every tested problem correctly, although individual attempts sometimes failed. This limits a broad reading of universal unreliability but does not contradict the authors' narrower account of their own tests.
- Publisher
- arXiv; Tanuj Kumar and Mikhail A. Kats
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Independent examination of dated original transcripts, or replication using the relevant historical model and tool versions, verifying the reported errors in both domains.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Getting GPT to work with external tools is harder than you think
Thus far we have seen consistently unreliable results, from math to fact-checking.
- Where
- paragraph 22
- Published
- 2023-10-23
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-getting-gpt-to-work-with-external-44d1f1-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-getting-gpt-to-work-with-external-44d1f1-a1.