“Math is hard” — if you are an LLM – and why that matters
We recorded 5 claims from this post. 1 was checked against evidence. 4 can’t be checked. 4 more are held. Read the post at its source.
Claims from this post (5)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Observation
“The paper turns out to be pretty great, doing exactly the right test, but it doesn’t prove what its title alleges.”
Our reading: The MathGLM paper does not prove its titular claim that GPT can solve mathematical problems without a calculator.
No horizon Evidence as of
Limitation: The reported accuracy does not establish guaranteed correctness, and the reported errors do not establish an absence of calculator-free capability.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“And 70% would be great for free throws; for multiplication it sucks.”
Our reading: Around 70% accuracy is unacceptable for multiplication, which requires exact answers.
No horizon
Assessment: UnresolvedSaid
Observation
“The LLM never induces such an algorithm.”
Our reading: The LLM does not induce an algorithm that actually computes multiplication.
No horizon Evidence as of
Limitation: This does not establish that MathGLM learned a general multiplication algorithm.
Assessment: Can’t be checkedSaid
Forecast
“Some kind of hybrid may well work, but LLMs on their own remain stuck.”
Our reading: Some kind of hybrid system may well succeed at reliable math where pure LLMs fail.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“That, in a nutshell, is why we should never trust pure LLMs; even under carefully controlled circumstances with massive amounts of directly relevant data, they still never really get even the most basic linear functions.”
Our reading: Pure LLMs should never be trusted.
No horizon
Held from this post (4)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The record does not meet the current rules. Our reading, not assessed: LLMs on their own remain stuck, unable to reliably learn abstractions like multiplication.
- The two reviews reached different assessments. Our reading, not assessed: MathGLM's performance steadily declines as the problems get bigger, with its gap to a calculator steadily increasing, and the LLM-based system generalizes by similarity, doing better on cases in or near the training set and never, ever getting to a complete, abstract, reliable representation of what multiplication is.
- The criteria are ambiguous. Our reading, not assessed: Even with controlled conditions and massive relevant data, pure LLMs never truly learn even the most basic linear functions.
- The record does not meet the current rules. Our reading, not assessed: The weakness of neural networks in learning abstract, similarity-independent rules, identified in the early 1990s, still persists in current deep learning.