“And we have yet to see any evidence that any kind of prompt helps in that regard.”
Our reading:
The authors have yet to see any evidence that any kind of prompt helps get AIs to say 'I give up' when they cannot solve a problem.
Assessment: Unresolved
What happened
Was it true for the period described?
Outcome: Unresolved
We checked, and the evidence so far doesn’t settle it.
Evidence up to . Checked .
Why
The explicit limit is what Davis and Marcus had seen. Research showing that prompting can improve abstention would not contradict this statement unless they had encountered relevant evidence before publication. The searches did not establish that exposure, and their own assertion cannot independently prove its absence. Replacing their statement with a claim that no research existed would change what they said.
Caveats
- Public searches cannot reconstruct private reading or correspondence.
- Evidence about general question-answering abstention may not address failure recognition in mathematical proofs.
Evidence (0 items)
Nothing we found settles this yet.
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Conditions
- as far as the authors have seen (their words)
Where we had to interpret
- We refers to Ernest Davis and Gary Marcus. The claim concerns evidence they had encountered by publication, not all publicly available research.
- In that regard refers to admitting inability to solve the mathematical proof problems under discussion instead of claiming an invalid solution.
- Helps means improves appropriate acknowledgment of failure compared with a relevant baseline; it does not require perfect abstention.
- The phrase have yet to see describes their experience through April 5, 2025, with no specified starting date.
Right if: Independently corroborated records of the authors' relevant evidence exposure establish that neither had encountered evidence of prompting improving appropriate admission of failure on these mathematical reasoning tasks by publication.
Wrong if: Dated records establish that, before publication, the authors had encountered relevant evidence showing such an improvement. Publication of a study alone is insufficient.
We wrote this test after the outcome was known.
What would settle it
Independently corroborated, dated records establishing the authors' exposure to relevant comparative prompting evidence before publication, or sufficiently complete records establishing its absence.
We found 13 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, Reports of LLMs mastering math have been greatly exaggerated, paragraph 16.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-28. The post may have been edited after 2025-04-05.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, Reports of LLMs mastering math have been greatly exaggerated
And we have yet to see any evidence that any kind of prompt helps in that regard.
- Where
- paragraph 16
- Published
- 2025-04-05
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Unresolved. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
A reviewer said our test wasn’t fair to what was actually said.