CONFIRMED: LLMs have indeed reached a point of diminishing returns
We recorded 13 claims from this post. 6 were checked against evidence. 7 can’t be checked. 5 more are held. Read the post at its source.
Claims from this post (13)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
Observation
“Just as I argued here in April 2024, LLMs have reached a point of diminishing returns.”
Our reading: LLMs have reached a point of diminishing returns from scaling.
No horizon Evidence as of
Limitation: The evidence lacks a comparable series of frontier training runs measuring capability gains against proportional increases in compute and data [e1, e2].
Assessment: Can’t be checkedSaid
Forecast
“The economics are likely to be grim.”
Our reading: The economics of LLMs are likely to be grim.
No horizon
Assessment: Can’t be checkedSaid
Conditional forecast
“When everyone realizes this, the financial bubble may burst quickly; even NVidia might take a hit, when people realize the extent to which its valuation was based on a false premise.”
Our reading: Once people realize LLM economics don't work, the AI financial bubble may burst quickly and even Nvidia might take a hit.
No horizon
Only if: when everyone realizes LLM economics don't work
Assessment: UnresolvedSaid
Forecast
“Given the cost of chips, profits will be elusive.”
Our reading: Because chips are costly, LLM companies will struggle to make profits.
No deadline stated Evidence as of
Limitation: The cited financial evidence concerns one developer.
Assessment: UnresolvedSaid
Forecast
“LLMs such as they are, will become a commodity; price wars will keep revenue low.”
Our reading: LLMs such as they are will become a commodity, and price wars will keep revenue low.
No deadline stated Evidence as of
Limitation: The market study is a preprint.
Assessment: Can’t be checkedSaid
Forecast
“LLMs will not disappear, even if improvements diminish, but the economics will likely never make sense: additional training is expensive, the more scaling, the more costly.”
Our reading: LLMs will not disappear even if improvements diminish, but their economics will likely never make sense because additional training is expensive and more scaling is more costly.
No horizon
Qualifier: even if improvements diminish
Assessment: SupportedSaid
Observation
“Many of us, including Melanie Mitchell, Subbarao Kambahapati, Emily Bender, Ernie Davis, etc. have been emphasizing for ages that there are principled limits with LLMs.”
Our reading: Marcus and many others, including Melanie Mitchell, Subbarao Kambhampati, Emily Bender and Ernie Davis, have long emphasized that there are principled limits with LLMs.
No horizon Evidence as of
Limitation: 'For ages' is colloquial, and the evidence does not establish an identical duration for every researcher.
Assessment: UnresolvedSaid
Observation
“Meanwhile, precious little investment has been made in other approaches.”
Our reading: Very little investment has gone into AI approaches other than LLMs.
No horizon Evidence as of
Limitation: The observation period is an explicit annual-reporting interpretation.
Assessment: Can’t be checkedSaid
Conditional forecast
“If LLMs won’t get the US to trustworthy AI, and our adversaries invest in alternative approaches, we could easily be outfoxed.”
Our reading: If LLMs don't lead to trustworthy AI and adversaries invest in alternatives, the US could easily be outfoxed.
No horizon
Only if: LLMs won't lead to trustworthy AI
Assessment: Can’t be checkedSaid
Forecast
“The US has been putting all its AI eggs in the LLM basket, and that may well prove to be an epic, massive mistake.”
Our reading: The US has been putting all its AI eggs in the LLM basket, and that may well prove to be an epic, massive mistake.
No horizon
Qualifier: may well
Assessment: UnresolvedSaid
Conditional forecast
“If enthusiasm for GenAI dwindles and market valuations plummet, AI won’t disappear, and LLMs won’t disappear; they will still have their place as tools for statistical approximation.”
Our reading: If enthusiasm for GenAI dwindles and market valuations plummet, AI and LLMs will not disappear, and LLMs will still have their place as tools for statistical approximation.
No deadline stated Evidence as of
Only if: if enthusiasm for GenAI dwindles and market valuations plummet
Assessment: Can’t be checkedSaid
Forecast
“But that place may be smaller; it is entirely possible that LLMs on their own will never live up to last year’s wild expectations.”
Our reading: LLMs' role may be smaller than hoped, and LLMs on their own may never live up to the wild expectations of 2023.
No horizon
Qualifier: LLMs on their own
Assessment: Can’t be checkedSaid
Forecast
“Reliable, trustworthy AI is surely achievable, but we may need to go back to the drawing board to get there.”
Our reading: Reliable, trustworthy AI is achievable, but getting there may require new approaches beyond current LLMs.
No horizon
Held from this post (5)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews reached different assessments. Our reading, not assessed: Everyone is landing in more or less the same place, which leaves nobody with a moat.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews read the claim differently