Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
We recorded 31 claims from this post. 19 were checked against evidence. 12 can’t be checked. 16 more are held. Read the post at its source.
Claims from this post (31)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Forecast
“But I think a fair case can be made that it is not what it has often been cracked up to be, and probably never will be.”
Our reading: Marcus thinks a fair case can be made that ChatGPT is not what it has often been cracked up to be, and that it probably never will be.
No horizon
Qualifier: I think a fair case can be made
Assessment: UnresolvedSaid
Forecast
“I just don’t happen to think large language models like ChatGPT will get us there.”
Our reading: Large language models like ChatGPT will not get us to AGI.
No deadline stated Evidence as of
Limitation: AGI and the architectural boundary are not precisely defined.
Assessment: UnresolvedSaid
Forecast
“I think that the recurring core technical problems that we have seen (as discussed below) with LLMs aren’t going way; instead they inherent to the technology.”
Our reading: The recurring core technical problems of LLMs are inherent to the technology and will not go away.
No deadline stated Evidence as of
Limitation: The formal and practical meanings of hallucination differ.
Assessment: UnresolvedSaid
Observation
“And I certainly don’t think that ChatGPT has lived up to expectations.”
Our reading: ChatGPT has not lived up to expectations.
No horizon Evidence as of
Limitation: The capability comparison is company-reported and limited to its internal benchmark.
Assessment: MixedSaid
Observation
“productivity studies sometimes show a 30% gain, but none have come close to 10x (1000% gain).”
Our reading: No productivity studies of LLMs have shown anything close to a 10x gain; some show around 30%.
No horizon Evidence as of
Limitation: The counterexample involved six fictional cases and document production, not overall physician productivity.
Assessment: Can’t be checkedSaid
Observation
“Return on investment for corporate end users is also lower than many people must have imagined.”
Our reading: Return on generative AI investment for corporate end users is lower than many people must have imagined.
No horizon Evidence as of
Limitation: This does not assess the separate factual claims about MIT or McKinsey.
Assessment: UnresolvedSaid
Observation
“But time has shown that its results have held strong.”
Our reading: The MIT finding that only 5% of companies get a return on generative AI investment has held up.
No horizon Evidence as of
Limitation: Similar difficulties scaling AI do not independently corroborate a 5% return prevalence.
Assessment: Can’t be checkedSaid
Causal claim
“And worse, the economy itself has become so wrapped up in generative AI and its promises, that the economy itself is, by many accounts in serious jeopardy.”
Our reading: The economy has become so wrapped up in generative AI and its promises that, by many accounts, it is in serious jeopardy.
No horizon
Qualifier: by many accounts
Assessment: UnresolvedSaid
Conditional forecast
“If the economy goes down, ChatGPT will be at the center of the mess.”
Our reading: If the economy goes down, ChatGPT will be at the center of the mess.
No deadline stated Evidence as of
Only if: if the economy goes down
Assessment: SupportedSaid
Causal claim
“The results are disappointing because the underlying tech is unreliable, And that’s been obvious from the start.”
Our reading: Generative AI results are disappointing because the underlying technology is unreliable, which has been obvious from the start.
No horizon Evidence as of
Limitation: The studies cover particular occupations and deployments.
Assessment: UnresolvedSaid
Observation
“Three years and nearly a trillion dollars later almost of all it remains true. And it’s not just that GPT-4 was pretty much as I anticipated but that the same problems have continued to plague every single model since”
Our reading: Almost all of his December 2022 predictions about GPT-4 remain true, and the same problems plague every model since, including GPT-5, Claude, Gemini, Grok, Llama and DeepSeek.
No horizon Evidence as of
Limitation: Representative failures cannot establish a universal claim about all model versions.
Assessment: SupportedSaid
Observation
“And LLMs themselves have already resorted to using lots of external tools like calculators and python interpreters; it is already clear that they are more of a partial solution than the full solution people imagine.”
Our reading: LLMs have already resorted to using lots of external tools such as calculators and Python interpreters, and it is already clear that they are more of a partial solution than the full solution people imagine.
No horizon Evidence as of
Limitation: Tool adoption does not by itself prove necessity, failure, or a permanent capability ceiling.
Assessment: UnresolvedSaid
Causal claim
“And because of the discrepancy between reliability and the immense costs that stem from the inherent inefficiency of a system that it dependent on absorbing internet-scale data, it has thus far failed to make profits for companies like OpenAI.”
Our reading: Because of unreliability and immense costs from inefficiency, ChatGPT has failed so far to make profits for companies like OpenAI.
No horizon Evidence as of
Limitation: No matching January–June 2025 net-income statement was obtained; the claim's accounting basis remains unknown.
Assessment: Can’t be checkedSaid
Conditional forecast
“If things do go belly-up, and the whole economy falls into recession, the single biggest culprit in my mind, will be ChatGPT’s human avatar, its bullshit-spewing CEO Sam Altman, who hyped GPT-5 endlessly for years, pretending he knew that AGI was coming when in hindsight he was bluffing.”
Our reading: If things go belly-up and the whole economy falls into recession, the single biggest culprit in Marcus's mind will be Sam Altman, whom he describes as having hyped GPT-5 for years while bluffing about knowing that AGI was coming.
No horizon
Only if: if things do go belly-up
Assessment: UnresolvedSaid
Observation
“Worse, it doesn’t prove that its premise—that exponentially increasing compute—will actually bring us to artificial general intelligence. He’s conflating costs going up exponentially with benefits going up exponentially.”
Our reading: Jensen Huang's claim of exponential compute demand does not show that exponentially increasing compute will bring AGI; he conflates exponential costs with exponential benefits.
No horizon Evidence as of
Limitation: This assesses the referenced argument, not the eventual feasibility of AGI.
Assessment: UnresolvedSaid
Conditional forecast
“And if Sustekver and I (and at this point many others) are right that scaling is largely played out, a decline in demand is sure to follow.”
Our reading: If scaling is largely played out, a decline in demand for AI compute is sure to follow.
No deadline stated Evidence as of
Only if: if scaling is largely played out
Assessment: Can’t be checkedSaid
Forecast
“GenAI is like a rerun of that situation, an order or two of magnitude more hype, coupled with equal naïveté about what a cognitively adequate solution would actually look like, only this time with an extra decimal place, potentially a trillion dollars down the drain — and perhaps with far more collateral damage to the economy.”
Our reading: Generative AI is like a rerun of the driverless-car hype, with an order or two of magnitude more hype and equal naïveté about what a cognitively adequate solution would look like, potentially with a trillion dollars down the drain and perhaps far more collateral damage to the economy.
No horizon
Qualifier: potentially
Assessment: Can’t be checkedSaid
Rhetoric
“Anyone assuming that everything is simultaneously rising exponentially, and that it will continue to do so indefinitely, is a fool.”
Our reading: Anyone assuming that everything is simultaneously rising exponentially and will continue to do so indefinitely is a fool.
No horizon
Assessment: UnresolvedSaid
Causal claim
“Too much industry hype is repeated with too little skepticism, driving up market valuations on analyses that often feel to me superficial.”
Our reading: Too much industry hype is repeated with too little skepticism, driving up market valuations on analyses that often feel to Marcus superficial.
No horizon Evidence as of
Limitation: Correlation between optimistic coverage and rising prices would not establish the direction of causation.
Assessment: Can’t be checkedSaid
Observation
“The market is maybe finally catching on.”
Our reading: The market may finally be recognizing AI's limits, as shown by falls in Nvidia, CoreWeave and Oracle stock.
No horizon
Assessment: Can’t be checkedSaid
Observation
“OpenAI itself isn’t traded on the open market, but if it had, it’s likely that it, too, would have fallen over the last month, for all the reasons I described, and one more: they don’t have moat.”
Our reading: OpenAI has no moat, and its value would likely have fallen over the last month if it were publicly traded.
No horizon
Assessment: UnresolvedSaid
Observation
“The reality is OpenAI has ridden to a $500 billion valuation on hopes, not profits. And those hopes are slipping away, as usage declines and others catch up.”
Our reading: OpenAI's $500 billion valuation rests on hopes rather than profits, and those hopes are slipping as usage declines and competitors catch up.
No horizon Evidence as of
Limitation: October 2025 is the contract's selected reporting period; the claim itself specifies no usage window.
Assessment: UnresolvedSaid
Forecast
“In the end LLMs will continue to exist, no matter what; they will get cheaper. And they will find some utility (indeed they are already have).”
Our reading: LLMs will continue to exist, get cheaper, and have some utility.
No deadline stated Evidence as of
Limitation: The deployment and efficiency evidence is vendor-reported.
Assessment: UnresolvedSaid
Forecast
“But they will never be AGI, and they never be much more than commodity.”
Our reading: LLMs will never be AGI and will never be much more than a commodity.
No deadline stated Evidence as of
Limitation: One benchmark cannot settle AGI.
Assessment: Can’t be checkedSaid
Observation
“In many domains. answers might be like 80% correct and 20% absurd, but without a whole lot of task-specific engineering required.”
Our reading: In many domains ChatGPT's answers may be roughly 80% correct and 20% absurd, with little task-specific engineering.
No horizon
Qualifier: in many domains
Assessment: UnresolvedSaid
Observation
“Ultimately though, what LLMs are great at seeming is plausible – while being much less good at getting their facts straight.”
Our reading: LLMs are good at seeming plausible but much less good at getting facts right.
No horizon Evidence as of
Limitation: The located plausibility experiment concerns prompted rationales rather than ordinary factual answers.
Assessment: UnresolvedSaid
Observation
“In some ways, GPT-5 is more capable than most 7 or 8 year olds.”
Our reading: In some ways, such as trivia and writing python code, GPT-5 is more capable than most 7- or 8-year-olds.
No horizon Evidence as of
Limitation: The available model evidence is company-reported and predates the target period.
Assessment: UnresolvedSaid
Observation
“More than that, any normal seven-year old develops a rich model of the world, including the objects they see, the people they interact with and so on, and even how their bodies work. GPT is still just faking it.”
Our reading: Unlike a seven-year-old, GPT lacks a rich model of the world and is just faking it.
No horizon Evidence as of
Limitation: The behavioral reading of 'faking it' is an evaluator interpretation.
Assessment: Can’t be checkedSaid
Observation
“People are now very actively trying to put variations of large language models into humanoid robots, but I can’t imagine any could do what my daughter was able to do then, let alone what she can do nowadays as a curious, precocious eleven-year-old.”
Our reading: People are very actively trying to put LLM variants into humanoid robots, but Marcus cannot imagine any of them could do what his daughter did at three (inventing a new way to climb out of her chair), let alone what she can do now at eleven.
No horizon
Qualifier: I can't imagine any could
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“For the love of Darwin, please let’s spend the next eight years considering new approaches, not tanking the economy on more of the same.”
Our reading: The AI field should spend the next eight years on new approaches rather than risking the economy on more LLM scaling.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“ChatGPT has had a good (though not great) run; it’s time for something new.”
Our reading: ChatGPT has had a good but not great run, and the field should move to something new.
No horizon
Held from this post (16)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: GPT-5 has not solved the core LLM problems, as many people imagined it would; had it done so, it would be of enormous economic value.
- The two reviews reached different assessments. Our reading, not assessed: Census Bureau data already shows signs of generative AI demand tapering off, which is not consistent with a genuine exponential.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: Google, with Gemini 3, appears to have overtaken OpenAI in the last month, beating GPT-5 on many benchmarks.
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: LLMs have become a commodity, profits outside Nvidia are scarce, and OpenAI is losing billions of dollars a month.
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: So far as Marcus knows, even the latest language models are still powerful but hard to control, still can't reason reliably, still don't work reliably with external tools, continue to hallucinate, still can't match domain-specific models, and continue to struggle with alignment.
- The record does not meet the current rules. Our reading, not assessed: Across hundreds of LLMs, the same failure modes always appear, a pattern Marcus calls astonishingly clear.
- The two reviews reached different assessments. Our reading, not assessed: ChatGPT obviously lacks the reliability of calculators, and its hallucinations and occasional bizarre errors should be shocking given that, unlike seven-year-olds, LLMs have swallowed the entire internet.
- The record does not meet the current rules. Our reading, not assessed: GPT-5 is more impressive than the original ChatGPT (GPT-3.5), the latest models have more utility, and LLMs have certainly come a long way.
- The criteria are ambiguous. Our reading, not assessed: Amateurs can build prototypes with AI coding, but the generated code tends to be shaky and few can make it robust.
- The two reviews reached different assessments. Our reading, not assessed: Because you never know what ChatGPT will produce, you can never really trust it.
- The two reviews reached different assessments. Our reading, not assessed: Scaling has not come close to solving the reliability, world-understanding and hallucination problems of ChatGPT.