Claims checked against evidence by Gary Marcus, page 4 of 15
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Claims checked against evidence by Gary Marcus (1403)
Each of these claims was checked against dated evidence.
Page 4 of 15: claims 301 to 400 of 1403, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: UnresolvedSaid
Conditional forecast
“If we are to build AGI, we are going to need to learn something from humans, how they reason and understand the physical world, and how they represent and acquire language and complex concepts.”
Our reading: If we are to build AGI, we are going to need to learn something from humans: how they reason and understand the physical world, and how they represent and acquire language and complex concepts.
No deadline stated Evidence as of
Only if: if we are to build AGI
Source: “Scale Is All You Need” is dead
Assessment: SupportedSaid
Observation
“The machine learning community, which has unwelcomed me since I started vocally critiquing the scaling approach, has finally started to come around.”
Our reading: The machine learning community has started to accept critiques of the scaling approach.
No horizon Evidence as of
Limitation: Rests on one industry write-up plus individual statements. No measured trend.
Source: “Scale Is All You Need” is dead
Assessment: SupportedSaid
Observation
“And now, at last, the field is starting to realize it.”
Our reading: The AI field is now starting to realize that scaling alone will not reach AGI.
No horizon Evidence as of
Limitation: No earlier survey is available for comparison, so the growth of this view is inferred rather than measured.
Source: “Scale Is All You Need” is dead
Assessment: UnresolvedSaid
Observation
“Worse, they are burning billions of dollars of month.”
Our reading: OpenAI is burning billions of dollars per month.
No horizon Evidence as of
Limitation: The first-half and full-year reports do not establish a common accounting basis or isolate July–September cash burn; subtracting their figures cannot settle the claim [e1, e2].
Source: OpenAI’s “Code Red”
Assessment: UnresolvedSaid
Forecast
“In the end LLMs will continue to exist, no matter what; they will get cheaper. And they will find some utility (indeed they are already have).”
Our reading: LLMs will continue to exist, get cheaper, and have some utility.
No deadline stated Evidence as of
Limitation: The deployment and efficiency evidence is vendor-reported.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Conditional forecast
“And if Sustekver and I (and at this point many others) are right that scaling is largely played out, a decline in demand is sure to follow.”
Our reading: If scaling is largely played out, a decline in demand for AI compute is sure to follow.
No deadline stated Evidence as of
Only if: if scaling is largely played out
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“More than that, any normal seven-year old develops a rich model of the world, including the objects they see, the people they interact with and so on, and even how their bodies work. GPT is still just faking it.”
Our reading: Unlike a seven-year-old, GPT lacks a rich model of the world and is just faking it.
No horizon Evidence as of
Limitation: The behavioral reading of 'faking it' is an evaluator interpretation.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Forecast
“But they will never be AGI, and they never be much more than commodity.”
Our reading: LLMs will never be AGI and will never be much more than a commodity.
No deadline stated Evidence as of
Limitation: One benchmark cannot settle AGI.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“In some ways, GPT-5 is more capable than most 7 or 8 year olds.”
Our reading: In some ways, such as trivia and writing python code, GPT-5 is more capable than most 7- or 8-year-olds.
No horizon Evidence as of
Limitation: The available model evidence is company-reported and predates the target period.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“Ultimately though, what LLMs are great at seeming is plausible – while being much less good at getting their facts straight.”
Our reading: LLMs are good at seeming plausible but much less good at getting facts right.
No horizon Evidence as of
Limitation: The located plausibility experiment concerns prompted rationales rather than ordinary factual answers.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“The reality is OpenAI has ridden to a $500 billion valuation on hopes, not profits. And those hopes are slipping away, as usage declines and others catch up.”
Our reading: OpenAI's $500 billion valuation rests on hopes rather than profits, and those hopes are slipping as usage declines and competitors catch up.
No horizon Evidence as of
Limitation: October 2025 is the contract's selected reporting period; the claim itself specifies no usage window.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“And I certainly don’t think that ChatGPT has lived up to expectations.”
Our reading: ChatGPT has not lived up to expectations.
No horizon Evidence as of
Limitation: The capability comparison is company-reported and limited to its internal benchmark.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Causal claim
“Too much industry hype is repeated with too little skepticism, driving up market valuations on analyses that often feel to me superficial.”
Our reading: Too much industry hype is repeated with too little skepticism, driving up market valuations on analyses that often feel to Marcus superficial.
No horizon Evidence as of
Limitation: Correlation between optimistic coverage and rising prices would not establish the direction of causation.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: SupportedSaid
Observation
“And LLMs themselves have already resorted to using lots of external tools like calculators and python interpreters; it is already clear that they are more of a partial solution than the full solution people imagine.”
Our reading: LLMs have already resorted to using lots of external tools such as calculators and Python interpreters, and it is already clear that they are more of a partial solution than the full solution people imagine.
No horizon Evidence as of
Limitation: Tool adoption does not by itself prove necessity, failure, or a permanent capability ceiling.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“Three years and nearly a trillion dollars later almost of all it remains true. And it’s not just that GPT-4 was pretty much as I anticipated but that the same problems have continued to plague every single model since”
Our reading: Almost all of his December 2022 predictions about GPT-4 remain true, and the same problems plague every model since, including GPT-5, Claude, Gemini, Grok, Llama and DeepSeek.
No horizon Evidence as of
Limitation: Representative failures cannot establish a universal claim about all model versions.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“Worse, it doesn’t prove that its premise—that exponentially increasing compute—will actually bring us to artificial general intelligence. He’s conflating costs going up exponentially with benefits going up exponentially.”
Our reading: Jensen Huang's claim of exponential compute demand does not show that exponentially increasing compute will bring AGI; he conflates exponential costs with exponential benefits.
No horizon Evidence as of
Limitation: This assesses the referenced argument, not the eventual feasibility of AGI.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Conditional forecast
“If the economy goes down, ChatGPT will be at the center of the mess.”
Our reading: If the economy goes down, ChatGPT will be at the center of the mess.
No deadline stated Evidence as of
Only if: if the economy goes down
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Forecast
“I think that the recurring core technical problems that we have seen (as discussed below) with LLMs aren’t going way; instead they inherent to the technology.”
Our reading: The recurring core technical problems of LLMs are inherent to the technology and will not go away.
No deadline stated Evidence as of
Limitation: The formal and practical meanings of hallucination differ.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Observation
“But time has shown that its results have held strong.”
Our reading: The MIT finding that only 5% of companies get a return on generative AI investment has held up.
No horizon Evidence as of
Limitation: Similar difficulties scaling AI do not independently corroborate a 5% return prevalence.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: SupportedSaid
Causal claim
“The results are disappointing because the underlying tech is unreliable, And that’s been obvious from the start.”
Our reading: Generative AI results are disappointing because the underlying technology is unreliable, which has been obvious from the start.
No horizon Evidence as of
Limitation: The studies cover particular occupations and deployments.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: MixedSaid
Observation
“productivity studies sometimes show a 30% gain, but none have come close to 10x (1000% gain).”
Our reading: No productivity studies of LLMs have shown anything close to a 10x gain; some show around 30%.
No horizon Evidence as of
Limitation: The counterexample involved six fictional cases and document production, not overall physician productivity.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: UnresolvedSaid
Forecast
“I just don’t happen to think large language models like ChatGPT will get us there.”
Our reading: Large language models like ChatGPT will not get us to AGI.
No deadline stated Evidence as of
Limitation: AGI and the architectural boundary are not precisely defined.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: SupportedSaid
Observation
“The unfairly dismissed Apple reasoning paper laid bare the generalization issues; another paper called “Is Chain-of-Thought Reasoning of LLMs Mirage? A Data Distribution Lens” put a further nail in the LLM reasoning and generalization coffin.”
Our reading: Research papers (Apple's reasoning paper and the chain-of-thought mirage paper) demonstrated LLM reasoning and generalization failures.
No horizon Evidence as of
Limitation: The result depends on reading the claim as empirical task failures rather than universal incapacity.
Assessment: UnresolvedSaid
Observation
“The first and most obvious answer is money, which I estimate, back of the envelope as (roughly) a trillion dollars, much of it on Nvidia chips and massive salaries.”
Our reading: Supposing for the sake of argument that LLMs have run their course and new ideas are needed, Marcus estimates, back of the envelope, the money cost of the field taking so long to recognize this at roughly a trillion dollars, much of it on Nvidia chips and massive salaries.
No horizon Evidence as of
Limitation: The spending boundary and starting date remain ambiguous.
Assessment: SupportedSaid
Observation
“Each new bigger, more expensive model ekes out measurable improvements, but returns appear to be diminishing (that’s what Sutskever is saying about the Kaplan laws) and none of these experiments has solved core issues around hallucinations, generalization, planning and reasoning, as Sutskever too now recognizes.”
Our reading: No LLM scaling experiment has solved the core problems of hallucination, generalization, planning and reasoning.
No horizon Evidence as of
Limitation: 'Solved' has no stated threshold. Someone who counts IMO gold or big benchmark gains as solving reasoning would read the claim differently.
Assessment: SupportedSaid
Observation
“It has never worked.”
Our reading: Building ever larger LLMs has never succeeded in reaching AGI.
No horizon Evidence as of
Limitation: The conclusion depends on the ordinary public-demonstration reading of AGI.
Assessment: SupportedSaid
Causal claim
“A big part of the culprit? Maximizing metrics for user engagement.”
Our reading: OpenAI maximizing user-engagement metrics was a big part of the cause of ChatGPT users having mental health crises.
No horizon Evidence as of
Limitation: Causation rests on investigative reporting and company statements. No clinical study or court finding in the packet establishes that engagement-tuned updates caused specific users' crises.
Source: ChatGPT and delusions: an important new inside look at OpenAI
Assessment: UnresolvedSaid
Forecast
“AGI ain’t actually coming in 2027, and even Kokotajlo, known for his aggressive timelines, is now with me in thinking we probably won’t see AGI this decade.”
Our reading: AGI will not arrive in 2027.
Not yet due Evidence as of
Limitation: The AGI criterion is an ordinary-language interpretation rather than a test Marcus specified.
Source: BREAKING: The AI 2027 doomsday scenario has officially been postponed
Assessment: UnresolvedSaid
Forecast
“AGI is NOT in fact coming in Trump’s second term, and never was.”
Our reading: AGI will not arrive during Trump's second term (ending January 2029).
Not yet due Evidence as of
Limitation: No single benchmark establishes the broad AGI criterion.
Source: BREAKING: The AI 2027 doomsday scenario has officially been postponed
Assessment: UnresolvedSaid
Observation
“Suffice for now to say that we have built (and continue to build) our entire national policy (such as it is) around an idea that was never grounded in a sound analysis of AI.”
Our reading: The US has built, and continues to build, its entire national policy (such as it is) around an idea, in context the AI 2027 picture of imminent AGI, that was never grounded in a sound analysis of AI.
No horizon Evidence as of
Limitation: The packet lacks policy-development records and the technical analyses actually relied upon during January 20–November 21, 2025.
Source: BREAKING: The AI 2027 doomsday scenario has officially been postponed
Assessment: SupportedSaid
Observation
“Google announced their big new state-of-the-art model yesterday, Gemini 3, on most measures better than any other model out there.”
Our reading: Gemini 3 is better than any other model on most measures.
No horizon Evidence as of
Limitation: Much of the comparison comes from Google's own chosen benchmarks.
Source: Hot take on Google’s Gemini 3
Assessment: ContradictedSaid
Observation
“(It’s only when ChatGPT eclipsed Meta that LeCun began to be sharply, publicly critical of LLMs)”
Our reading: LeCun began to be sharply and publicly critical of LLMs only when ChatGPT eclipsed Meta.
No horizon Evidence as of
Limitation: The decisive essay was coauthored.
Assessment: SupportedSaid
Observation
“But he neither invented them nor was he the first to apply the back-propagation algorithm to learning their weights (though many people mistakenly believe this).”
Our reading: LeCun neither invented convolutional neural networks nor was the first to train their weights with back-propagation.
No horizon Evidence as of
Limitation: This result includes temporal convolutional networks; a claim restricted to two-dimensional image CNNs would require a separate priority comparison.
Assessment: SupportedSaid
Observation
“Oracle debt downgraded.”
Our reading: Oracle's debt has been downgraded, which is an ominous sign for generative AI.
No horizon Evidence as of
Limitation: The evidence is secondary coverage of the Barclays note, not the note itself.
Assessment: SupportedSaid
Observation
“Coreweave drops 20% in a week.”
Our reading: CoreWeave's stock fell 20% in one week, which is an ominous sign for generative AI.
No horizon Evidence as of
Limitation: The weekly figure comes from a news report, not a primary price-data feed.
Assessment: SupportedSaid
Observation
“Meta financing games revealed.”
Our reading: Meta's financing games have been revealed, presented as one of five recent ominous signs for generative AI.
No horizon Evidence as of
Limitation: The transaction identification and one-month lookback are evaluator interpretations.
Assessment: SupportedSaid
Observation
“Softbank sells entire Nvidia position.”
Our reading: SoftBank has sold its entire Nvidia position, which is an ominous sign for generative AI.
No horizon Evidence as of
Limitation: The 'ominous sign' framing is not scored.
Assessment: UnresolvedSaid
Observation
“In short, Altman was—likely in conjunction with Nvidia, which also just seemed to be laying groundwork for a bailout—launching a full court press for loan guarantees when he got caught with hands in the cookie jar.”
Our reading: Altman was launching a full court press for federal loan guarantees, likely in conjunction with Nvidia, which also seemed to be laying groundwork for a bailout.
No horizon Evidence as of
Limitation: The metaphor leaves the required intensity imprecise.
Assessment: UnresolvedSaid
Observation
“But it is no longer about lying to employees. It is about directly lying to the American public.”
Our reading: Sam Altman is now directly lying to the American public, not just to employees.
No horizon Evidence as of
Limitation: The submission does not expressly request guarantees for OpenAI's own data-center borrowing [e2].
Assessment: UnresolvedSaid
Conditional forecast
“Which means you, the taxpayer, will be footing the bill.”
Our reading: If the government goes along, taxpayers will end up paying for AI companies' borrowing.
No deadline stated Evidence as of
Only if: if the government goes along with the backstop
Source: OpenAI probably can’t make ends meet. That’s where you come in.
Assessment: MixedSaid
Observation
“What Altman couldn’t say then was that he has a plan, to reduce the cost of his borrowing … by having the American taxpayer (indirectly) foot the bill.”
Our reading: OpenAI plans to reduce its borrowing costs by having American taxpayers indirectly bear the cost.
No horizon Evidence as of
Limitation: The original remarks and clarification are reproduced by news outlets; a complete primary transcript was not obtained.
Source: OpenAI probably can’t make ends meet. That’s where you come in.
Assessment: UnresolvedSaid
Observation
“Nonetheless, a very large fraction of our economy is going into the world of price wars and little moat, massive, massive investments into companies that are all basically following the same strategy of investing more and more money into one single idea, viz. scaling ever larger language models, in hopes that something magical will emerge.”
Our reading: A very large fraction of the US economy is going into massive investments in companies that are all basically following the same strategy of scaling ever larger language models, in hopes that something magical will emerge.
No horizon Evidence as of
Limitation: The broad equipment, software, R&D and data-center categories do not isolate investment in larger-language-model development.
Assessment: UnresolvedSaid
Observation
“Yet investments in the same single approach grow every day.”
Our reading: Investments in the single LLM/transformer approach continue to grow.
No horizon Evidence as of
Limitation: The evidence lacks a comparable aggregate series isolating LLM/transformer investment in Q3 2025 and earlier periods.
Assessment: UnresolvedSaid
Observation
“Modest profits was maybe a bit optimistic, with most LLM developers losing money.”
Our reading: Most LLM developers are losing money.
No horizon Evidence as of
Limitation: The selected financial interval is an evaluator interpretation.
Assessment: SupportedSaid
Observation
“GPT-5 was disappointing, and didn’t come in 2024”
Our reading: GPT-5 was disappointing and was not released in 2024.
No horizon Evidence as of
Limitation: 'Disappointing' is evaluative; reactions were mixed, not unanimous.
Assessment: SupportedSaid
Observation
“Meanwhile we have entered an era, perhaps unprecedented, of extreme financial interdependence and circular investment:”
Our reading: The AI industry has entered a perhaps unprecedented era of extreme financial interdependence and circular investment.
No horizon Evidence as of
Limitation: Extreme is a qualitative judgment without a stated threshold; the result depends on the ordinary reading specified in the contract.
Assessment: SupportedSaid
Observation
“no robust solution to hallucinations”
Our reading: There is no robust solution to LLM hallucinations.
No horizon Evidence as of
Limitation: 'Robust' has no threshold; partial mitigations exist and rates are falling.
Assessment: UnresolvedSaid
Conditional forecast
“If and when the whole generative AI edifice comes crashing down, it will be in no small part because too many otherwise smart investors read too many graphs—from “LLM scaling” to usage statistics—too naively, assuming that things had been going up for a while would continue to go up at the same pace, indefinitely.”
Our reading: If generative AI collapses, a large part of the cause will be investors naively extrapolating scaling and usage trends.
No deadline stated Evidence as of
Only if: if and when the generative AI edifice comes crashing down
Assessment: UnresolvedSaid
Causal claim
“The problem, as always, lies in generalizing outside the training distribution.”
Our reading: The problem with vibe coding lies, as always, in generalizing outside the training distribution.
No horizon Evidence as of
Limitation: Qualitative reports do not measure causal effects.
Source: Is vibe coding dying?
Assessment: UnresolvedSaid
Forecast
“Remember how in October and in March I told you that vibe coding — in the sense of amateurs using large language models to write code to “build products that would have previously required teams of engineers” — would never be remotely reliable?”
Our reading: Vibe coding by amateurs using LLMs to build products that previously required engineering teams will never be remotely reliable.
No deadline stated Evidence as of
Limitation: Most of the evidence is anecdotal or comes from a vendor's CEO.
Source: Is vibe coding dying?
Assessment: SupportedSaid
Observation
“All that really happened is that GPT-5 (oversimplifying slightly) crawled the web for solutions to already-solved problems.”
Our reading: GPT-5 did not solve open Erdős problems; it essentially located existing solutions to already-solved problems on the web.
No horizon Evidence as of
Limitation: The result depends on treating the crawler wording as the acknowledged simplification.
Source: Erdosgate
Assessment: UnresolvedSaid
Observation
“I found that the claim of “solution” was pretty misleading, as I summed up in a tweet, and they refused to correct their misleading presentation:”
Our reading: In 2019 OpenAI's claim that its robot 'solved' the Rubik's cube was misleading, and OpenAI refused to correct it.
No horizon Evidence as of
Limitation: Disagreement over wording does not by itself prove a factual misrepresentation.
Source: Erdosgate
Assessment: UnresolvedSaid
Observation
“June, 2025: the Apple reasoning paper confirmed that even with “reasoning”, LLMs still can’t solve distribution shift, the core Achille’s heel in neural networks that I have been writing about for nearly 30 years.”
Our reading: Even with reasoning, LLMs still cannot handle distribution shift, as the June 2025 Apple paper confirmed.
No horizon Evidence as of
Limitation: The competing papers are evidence of a scientific dispute, not equally decisive demonstrations.
Source: Game over. AGI is not imminent, and LLMs are not the royal road to getting there.
Assessment: UnresolvedSaid
Observation
“We aren’t close to AGI, but we are reshaping our world on the premise that we are.”
Our reading: We are not close to AGI.
No horizon Evidence as of
Limitation: Benchmark failures do not measure time remaining to AGI.
Source: The Grand AGI Delusion
Assessment: UnresolvedSaid
Observation
“And schoolchildren using ChatGPT to do their homework aren’t enough to cover that.”
Our reading: Schoolchildren using ChatGPT to do their homework are not enough to cover OpenAI's immense costs and cash burn.
No horizon Evidence as of
Limitation: The reporting window and meaning of 'cover that' are not explicit.
Source: OpenAI’s future, foretold?
Assessment: SupportedSaid
Observation
“GenAI certainly sounds amazing – but it hasn’t always worked well enough to justify the costs.”
Our reading: Generative AI has not always worked well enough to justify its costs.
No horizon Evidence as of
Limitation: Support rests on preliminary, largely self-reported and anonymized deployment research.
Source: OpenAI’s future, foretold?
Assessment: SupportedSaid
Observation
“The notion on which this Substack was founded in May 2022 — that LLM scaling would not bring us to AGI — has gone mainstream.”
Our reading: The view that LLM scaling will not bring AGI has become mainstream.
No horizon Evidence as of
Limitation: The survey was academically weighted and was not a representative industry poll.
Assessment: UnresolvedSaid
Forecast
“that LLM scaling would not bring us to AGI”
Our reading: Scaling LLMs will not produce AGI.
No deadline stated Evidence as of
Limitation: There is no universally accepted AGI test.
Assessment: SupportedSaid
Observation
“The concerns about potential copyright violations that Reid Southen and I expressed here in and at greater length in the IEEE Spectrum in January 2024 continue almost two years later to be a problem.”
Our reading: Copyright violation concerns with generative AI remain a problem as of September 2025.
No horizon Evidence as of
Limitation: A lawsuit establishes a live allegation, not that the defendant infringed.
Assessment: UnresolvedSaid
Forecast
“Perhaps I also no longer need to warn people that the idea that coding would disappear soon is bullshit.”
Our reading: Human coding will not disappear soon due to AI.
No deadline stated Evidence as of
Limitation: This leaves the forecast open because of its timing, not because the evidence shows human coding disappeared.
Assessment: SupportedSaid
Observation
“No model could have done this in 2022. The graphics are vastly better than they used to be.”
Our reading: Image generation graphics quality has improved vastly since 2022, to a level no 2022 model could reach.
No horizon Evidence as of
Limitation: The evidence does not show that no 2022 model could produce a sharp, mechanically correct image of a rider. The 2022 test found some good outputs.
Source: Spinal Tap meets John Malkovich
Assessment: UnresolvedSaid
Observation
“Watch the tuners come and go! So much for object permanence.”
Our reading: Grok's generated video failed to keep objects consistent, with the bass tuners appearing and disappearing.
No horizon Evidence as of
Limitation: No evidence items show what the video contains.
Source: Spinal Tap meets John Malkovich
Assessment: UnresolvedSaid
Causal claim
“Agency plus LLMs have already led multiple “vibe coders” to lose databases, with new reports coming in every day or two, stemming simply from the unreliability inherent in LLMs (that Gary has so often harped on).”
Our reading: Agency combined with LLMs has already led multiple vibe coders to lose databases, with new reports every day or two, and these losses stem simply from the unreliability built into LLMs.
No horizon Evidence as of
Limitation: The Cursor incident rests on a user allegation without execution logs or an independent postmortem [e2].
Assessment: UnresolvedSaid
Forecast
“The bad news is that although this one was stopped, many others won’t be. No one patch would be enough, or even a thousand; there are just too many variations on the overall theme”
Our reading: Many attacks on AI coding tools won't be stopped; not even a thousand patches would suffice given the number of variations.
No deadline stated Evidence as of
Limitation: The evidence concerns researcher demonstrations rather than a census of malicious attacks.
Assessment: MixedSaid
Observation
“Some customers even wanted the old models back. Almost nobody felt satisfied.”
Our reading: Almost no one was satisfied with GPT-5, and some customers wanted the old models back.
No horizon Evidence as of
Limitation: No representative satisfaction survey of GPT-5 users from the launch week was found, so the share of satisfied users is not known.
Assessment: UnresolvedSaid
Forecast
“But between the disappointments of GPT-5 and a new study from METR that shows that LLMs do markedly better on coding benchmarks than in real-word practice, I think it is safe to say that LLMs won’t lead the way.”
Our reading: LLMs will not be the approach that leads the way to AGI.
No deadline stated Evidence as of
Limitation: The first accepted AGI is an interpretation of 'lead the way', not a criterion Marcus explicitly supplied.
Assessment: SupportedSaid
Observation
“But between the disappointments of GPT-5 and a new study from METR that shows that LLMs do markedly better on coding benchmarks than in real-word practice, I think it is safe to say that LLMs won’t lead the way.”
Our reading: LLMs do markedly better on coding benchmarks than in real-world coding practice.
No horizon Evidence as of
Limitation: The study does not establish the same mismatch across all developers, tasks or LLMs; it allows for substantial productivity gains in other coding settings [e2].
Assessment: UnresolvedSaid
Causal claim
“Critically, as I argued at the end of June (and going back to 2019) LLMs never induce proper world models, which is why, for example, they still can’t even play chess reliably,1 and continue to make stupid, head-scratching errors with startling regularity.”
Our reading: LLMs never induce proper world models, which causes them to be unable to play chess reliably and to make frequent stupid errors.
No horizon Evidence as of
Limitation: The strongest intervention evidence concerns a specialist chess model.
Assessment: UnresolvedSaid
Causal claim
“ChatGPT gets them wrong because it is munging words together, not genuinely processing an (elementary) model of British PM history.”
Our reading: ChatGPT gets the British prime minister tasks wrong because it is munging words together rather than genuinely processing an elementary model of British PM history.
No horizon Evidence as of
Limitation: Evidence from Llama-2 cannot directly support or contradict a claim about GPT-5's internal processing.
Assessment: UnresolvedSaid
Observation
“They don’t induce abstract models of the world that they can reliably engage, even when those models are given to them on a silver platter (as with the rules of chess).”
Our reading: LLMs do not induce abstract world models they can reliably use, even when given the rules explicitly.
No horizon Evidence as of
Limitation: RuleArena tests practical regulations rather than chess.
Assessment: UnresolvedSaid
Observation
“Even with thinking mode activated, the output was filled with errors that could easily have been avoided, with a 5-second Wikipedia search that any sensible human could have performed.”
Our reading: GPT-5 in thinking mode produced a list of Canadian Prime Ministers full of easily avoidable errors.
No horizon Evidence as of
Limitation: The retrieved plain-text version of the post does not supply independently verifiable image provenance.
Assessment: ContradictedSaid
Observation
“They do not sanity check their own work.”
Our reading: LLMs do not sanity check their own work.
No horizon Evidence as of
Limitation: SCoRe used specialized training; its results do not establish that every model checks every answer [e1].
Assessment: MixedSaid
Observation
“Cherry-picked studies, benchmark-gaming, and now even vibe-coded graphs, with zero transparency about how systems work or how they have been trained; public science is in the rear view mirror.”
Our reading: AI has been marked by cherry-picked studies, benchmark-gaming and now even vibe-coded graphs, with zero transparency about how systems work or how they have been trained, and public science is in the rear view mirror.
No horizon Evidence as of
Limitation: The judgment depends on reading zero transparency as no substantive disclosure, rather than insufficient disclosure for reproduction.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“The real news is a breaking study from Arizona State University that fully vindicates what I have told you for nearly 30 years—and more recently what Apple told you—about the core weakness of LLMs: their inability to generalize broadly.”
Our reading: LLMs' core weakness is their inability to generalize broadly, as confirmed by an ASU study.
No horizon Evidence as of
Limitation: This assesses the August 5 version available when Marcus wrote.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“Driverless cars that still are only available in couple percent of the world’s cities.”
Our reading: Driverless cars are available in only a couple percent of the world's cities.
No horizon Evidence as of
Limitation: This does not establish that worldwide availability exceeded his estimate.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: SupportedSaid
Observation
“Many of their best people have left.”
Our reading: Many of OpenAI's best people have left the company.
No horizon Evidence as of
Limitation: Documented prominence does not establish an objective ranking of OpenAI's best employees.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“What the mainstream media mostly hasn’t told you yet is that a few days later, hardly anybody is buying Altman’s story.”
Our reading: A few days after launch, hardly anybody believes Altman's claims that GPT-5 is like a PhD-level expert.
No horizon Evidence as of
Limitation: The evidence is news reports of individual reactions, not representative data.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: SupportedSaid
Observation
“People had grown to expect miracles, but GPT-5 is just the latest incremental advance.”
Our reading: GPT-5 is merely an incremental advance, not a radical leap.
No horizon Evidence as of
Limitation: The result depends on reading the comparison as against the immediately preceding models.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“Even some of the tech bros are waking up to the reality that “AGI in 2027” was marketing, not reality.”
Our reading: Even some tech bros are waking up to the reality that 'AGI in 2027' was marketing, not reality.
No horizon Evidence as of
Limitation: The observation concerns public attitudes by August 2025, not whether AGI ultimately arrives in 2027.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“The challenge of parts and wholes in generative images that Ernest Davis and I discussed here in December fared no better.”
Our reading: GPT-5 shows no improvement on the parts-and-wholes problem in generated images.
No horizon Evidence as of
Limitation: The only systematic test predates GPT-5 and used ChatGPT-4o.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“Elon is moving faster. Anthropic and Google and many others are nipping at their heels. Their relationship with Microsoft has frayed.”
Our reading: Elon Musk is moving faster than OpenAI, Anthropic and Google and many others are nipping at OpenAI's heels, and OpenAI's relationship with Microsoft has frayed.
No horizon Evidence as of
Limitation: Release cadence, benchmark rank and rate of capability improvement are different measures.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: SupportedSaid
Observation
“And, indeed, much as I anticipated here two weeks ago, the problems I have been pointing out over the last quarter century still lingered.”
Our reading: GPT-5 still has the longstanding problems Marcus has identified, such as chess rule-following and world models.
No horizon Evidence as of
Limitation: The main evidence is a single demonstration, not a benchmark.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Observation
“It’s not like other systems are faring much better.”
Our reading: Other generative AI systems such as Grok are not faring much better than GPT-5.
No horizon Evidence as of
Limitation: The packet contains no verified primary comparison from the August 7–9 launch window.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: SupportedSaid
Observation
“He still prefers “enshittification” to “slop”, but the problem is both serious and ubiquitous no matter what you call it.”
Our reading: The problem of AI slop/enshittification is serious and ubiquitous.
No horizon Evidence as of
Limitation: The result depends on the contract's reading of 'serious' and 'ubiquitous', which are evaluative or hyperbolic words.
Source: Slopocalypse Now
Assessment: SupportedSaid
Observation
“All the big companies have in fact introduced agents.”
Our reading: All the big AI companies have introduced AI agents as of August 2025.
No horizon Evidence as of
Limitation: The company universe is undefined.
Assessment: UnresolvedSaid
Causal claim
“In practice, those mistakes and errors have greatly limited the utility agents like ChatGPT agent.”
Our reading: Mistakes and errors have greatly limited the utility of agents like ChatGPT agent.
No horizon Evidence as of
Limitation: The causal evidence consists of small, selected task demonstrations.
Assessment: SupportedSaid
Observation
“The unrelenting hallucinations problem frequently rears its ugly head:”
Our reading: Hallucinations frequently occur in AI agents.
No horizon Evidence as of
Limitation: No standard rate measure; 'frequently' is judged from repeated documented cases.
Assessment: UnresolvedSaid
Observation
“That sentiment that has only grown over the last several months.”
Our reading: The sentiment that AI agents' reality doesn't match the hype has grown from March to August 2025.
No horizon Evidence as of
Limitation: Trust and disappointment with hype are different measures.
Assessment: SupportedSaid
Causal claim
“As a consequence of their superficial understanding of what they’re being asked, current agents are grossly vulnerable to cyberattacks.”
Our reading: Current AI agents are grossly vulnerable to cyberattacks because of their superficial understanding of their tasks.
No horizon Evidence as of
Limitation: No study in the packet tests 'superficial understanding' as a cause directly. The causal link rests on how successful attacks worked and on one expert's explanation.
Assessment: UnresolvedSaid
Observation
“But, none, to my knowledge are reliable, except maybe in very narrow uses cases.”
Our reading: To his knowledge, none of the big companies' AI agents are reliable, except maybe in very narrow use cases.
No horizon Evidence as of
Limitation: The July reports do not provide representative, repeated trials or systematic rates of consequential errors and human intervention [e2, e3].
Assessment: SupportedSaid
Causal claim
“In fundamentally unreliable systems like LLMs, that means multiple chances for error. Soon or later, those errors catch up. That can and sometimes does lead to catastrophe.”
Our reading: Because LLMs are fundamentally unreliable, multi-step agentic tasks accumulate errors that sometimes lead to catastrophe.
No horizon Evidence as of
Limitation: The evidence is incident reports, not a measured rate of how often multi-step errors cause serious damage.
Assessment: SupportedSaid
Forecast
“Guardrails (a la ChatGPT) may be in place, but the guardrails will teeter between being too weak (beaten by “jailbreaks”) and too strong (rejecting some perfectly reasonable requests).”
Our reading: Guardrails like ChatGPT's may be in place for GPT-5, but they will teeter between being too weak (beaten by jailbreaks) and too strong (rejecting some perfectly reasonable requests).
Settled before its deadline Evidence as of
Limitation: The evaluations do not establish temporal oscillation, ordinary-user failure rates, or persistence in one unchanged deployment [e2, e3].
Assessment: UnresolvedSaid
Forecast
“For that, we will, I predict, need genuinely new architectures that incorporate explicit knowledge and world models at their very core.”
Our reading: Trustworthy AI will require genuinely new architectures with explicit knowledge and world models at their core.
No deadline stated Evidence as of
Limitation: Behavior on a bounded benchmark cannot establish broad trustworthiness.
Assessment: UnresolvedSaid
Forecast
“GPT-5 will still, like its predecessors, be a bull in a china shop, reckless and hard to control.”
Our reading: GPT-5, like its predecessors, will be reckless and hard to control.
No deadline stated Evidence as of
Limitation: METR's task suite does not measure the prevalence or consequences of unintended behavior across ordinary deployments, and its detection methods may miss reward hacking [e1].
Assessment: UnresolvedSaid
Forecast
“Without external aids it won’t be able beat Meta’s Cicero in Diplomacy; it won’t be able to drive a car reliably; it won’t be able to reliably guide a robot like Optimus to be anything like as versatile as Rosie the Robot.”
Our reading: Without external aids, GPT-5 will not be able to beat Meta's Cicero at Diplomacy, will not be able to drive a car reliably, and will not be able to reliably guide a robot like Optimus to be anything like as versatile as Rosie the Robot.
No deadline stated Evidence as of
Limitation: This compound claim cannot be supported from evidence about robotics alone.
Assessment: UnresolvedSaid
Forecast
“Its natural language output still won’t be something that one can reliably hook up to downstream programs; it won’t be something, for example, that you can simply and directly hook up to a database or virtual assistant, with predictable results.”
Our reading: GPT-5's natural language output will not be something one can reliably hook up to downstream programs, for example simply and directly to a database or virtual assistant, with predictable results.
No deadline stated Evidence as of
Limitation: Distinguishing natural language from plaintext code materially affects the assessment.
Assessment: UnresolvedSaid
Conditional forecast
“Trustworthy, general artificial intelligence, aligned with human values, will come, when it does, from systems that are more structured, with more built-in knowledge, and will incorporate at least some degree of explicit tools for reasoning and planning, as well as explicit knowledge, that are lacking in systems like GPT.”
Our reading: When trustworthy general AI aligned with human values comes, it will come from systems that are more structured, have more built-in knowledge, and incorporate at least some degree of explicit tools for reasoning and planning, as well as explicit knowledge, that systems like GPT lack.
No deadline stated Evidence as of
Only if: when it does come
Assessment: UnresolvedSaid
Forecast
“GPT-5 will not have reliable models of the things that it talks about that are accessible to external programmers in a way that reliably feeds downstream processes.”
Our reading: GPT-5 will not have reliable models of the things it talks about that external programmers can access in a way that reliably feeds downstream processes.
No deadline stated Evidence as of
Limitation: The meaning of an accessible subject model materially affects what evidence would count.
Assessment: UnresolvedSaid
Forecast
“In retrospectives written in 2043, intellectual historians will conclude that there was an initial overemphasis on large language models, and a gradual but critical shift of the pendulum back to more structured systems with deeper comprehension.”
Our reading: In retrospectives written in 2043, intellectual historians will conclude that there was an initial overemphasis on large language models and a gradual but critical shift of the pendulum back to more structured systems with deeper comprehension.
Not yet due Evidence as of
Limitation: Future assessment will require a transparent selection of relevant historians and retrospectives.
Assessment: UnresolvedSaid
Observation
“I was right on the money then.”
Our reading: Marcus's December 2022 predictions about GPT-4 were accurate.
No horizon Evidence as of
Limitation: The assessment reads the retrospective as referring to all seven predictions; its intended scope is not explicit.