Supported claims by Gary Marcus, page 2 of 6
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Supported claims by Gary Marcus (530)
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Page 2 of 6: claims 101 to 200 of 530, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Observation
“OpenClaw itself is basically a cascade of LLMs.”
Our reading: OpenClaw is basically a cascade of LLMs.
No horizon Evidence as of
Limitation: Cascade is informal terminology.
Source: OpenClaw (a.k.a. Moltbot) is everywhere all at once, and a disaster waiting to happen
Assessment: SupportedSaid
Observation
“Today’s swarms, now emerging, can coordinate fleets of synthetic personas—sometimes with persistent identities—and move in ways that are hard to distinguish from real communities.”
Our reading: Emerging AI bot swarms can coordinate fleets of synthetic personas that are hard to distinguish from real communities.
No horizon Evidence as of
Limitation: 'Hard to distinguish' has no measure. The evidence does not show how long platforms or ordinary users took to detect these networks.
Assessment: SupportedSaid
Observation
“Venture capital is already helping industrialize astroturfing: Doublespeed, backed by Andreessen Horowitz, advertises a way to “orchestrate actions on thousands of social accounts” and to mimic “natural user interaction” on physical devices so the activity appears human.”
Our reading: Venture capital is already helping industrialize astroturfing; for example, Andreessen Horowitz-backed Doublespeed advertises a way to orchestrate actions on thousands of social accounts and to mimic natural user interaction on physical devices so the activity appears human.
No horizon Evidence as of
Limitation: Historical advertising wording rests on accessible reporting because the primary website has changed and the original 404 Media report was only partially accessible.
Assessment: SupportedSaid
Observation
“And it’s just getting worse.”
Our reading: The problem of chatbot-associated deaths is getting worse.
No horizon Evidence as of
Limitation: The assessment counts reported cases and lawsuits. It does not use measured death rates, and the evidence includes no per-user rate.
Assessment: SupportedSaid
Observation
“And then there is the growing problem of chatbots apparently inducing delusions in humans, that Kashmir Hill and others have been documenting.”
Our reading: Chatbots apparently inducing delusions in humans is a growing problem.
No horizon Evidence as of
Limitation: The evidence covers documented cases, not true incidence. Awareness may inflate the trend.
Assessment: SupportedSaid
Observation
“By now, there have been several reports of chatbot-associated deaths (both suicide and murder), enough that wikipedia has started keeping track.”
Our reading: By now there have been several reports of chatbot-associated deaths, both suicides and murders, enough that Wikipedia has started keeping track of them.
No horizon Evidence as of
Limitation: The reported associations do not establish that chatbots caused the deaths; the hearing account and complaint record allegations [e2, e3].
Assessment: SupportedSaid
Observation
“That deal was OpenAI’s to lose. And they lost it.”
Our reading: OpenAI lost the Apple Siri generative AI deal to Google.
No horizon Evidence as of
Limitation: Support depends on lost meaning did not secure the opportunity.
Assessment: SupportedSaid
Observation
“Many, many people were disappointed with GPT-5, arguably among the most overhyped products in history.”
Our reading: Many, many people were disappointed with GPT-5, which was arguably among the most overhyped products in history.
No horizon Evidence as of
Limitation: Support applies to substantial disappointment, not the qualified historical ranking.
Assessment: SupportedSaid
Observation
“I warned that LLMs might plateau within a year or two, and they have, that OpenAI would be mired in a lot of litigation, and they are.”
Our reading: OpenAI is mired in a lot of litigation.
No horizon Evidence as of
Limitation: The evidence covers one consolidated litigation group, not OpenAI’s complete litigation portfolio.
Assessment: SupportedSaid
Observation
“Most importantly, other companies like Anthropic and Google caught up; xAI and various Chinese companies aren’t far behind.”
Our reading: Anthropic and Google have caught up with OpenAI, and xAI and Chinese companies are not far behind.
No horizon Evidence as of
Limitation: The result depends on the ordinary competitive-tier reading of caught up and not far behind.
Assessment: SupportedSaid
Observation
“I also warned that LLMs would continue to have a truthiness problem and they have.”
Our reading: LLMs have continued to have a truthfulness problem.
No horizon Evidence as of
Limitation: This establishes persistence, not a uniform error rate across models or tasks.
Assessment: SupportedSaid
Observation
“Update: This is consistent with the finding of the Remote Labor Index that AI could only do about 2.5% of jobs, reported recently by the Washington Post.”
Our reading: Per the Remote Labor Index, AI can do only about 2.5% of jobs, consistent with LLMs adding little quantifiable value.
No horizon Evidence as of
Limitation: The benchmark result does not establish the share of all occupations AI can perform or aggregate economic value from LLM use [e1].
Source: Let’s be honest, Generative AI isn’t going all that well
Assessment: SupportedSaid
Observation
“LLMs can still cannot be trusted:”
Our reading: LLMs still cannot be trusted.
No horizon Evidence as of
Limitation: The supplied AA-Omniscience abstract does not specify whether browsing was enabled [e1].
Source: Let’s be honest, Generative AI isn’t going all that well
Assessment: SupportedSaid
Observation
“Why LLMs just can’t stop hallucinating.”
Our reading: LLMs are unable to stop hallucinating.
No horizon Evidence as of
Limitation: The Vectara figures come from one benchmark of summarization faithfulness. They may not reflect other tasks or other ways of measuring factual accuracy.
Source: AI Cartoon of The Year — and five (re)readings for 2026
Assessment: SupportedSaid
Observation
“The technical problems are not new.”
Our reading: The core technical problems of LLMs are not new.
No horizon Evidence as of
Limitation: The evidence establishes continuity in broad failure categories, not identical causes across model generations.
Assessment: SupportedSaid
Observation
“This is in line with the July METR study showing that coding tools sometimes impair productivity rather than increasing it.”
Our reading: The July METR study showed that AI coding tools sometimes impair productivity rather than increase it.
No horizon Evidence as of
Limitation: The trial involved experienced developers working on familiar open-source projects.
Assessment: SupportedSaid
Observation
“But many others have found that LLMs are not quite as advertised.”
Our reading: Many others have found that LLMs are not quite as good as advertised.
No horizon Evidence as of
Limitation: The advertising comparison is qualitative and supplied by interpretation.
Assessment: SupportedSaid
Observation
“As anticipated here yesterday, Trump’s decision to allow Nvidia to sell H200s to China hasn’t gone over well.”
Our reading: Trump's decision to allow Nvidia to sell H200 chips to China has not been well received.
No horizon Evidence as of
Limitation: This assesses substantial backlash, not majority public opinion.
Source: Has the Chinese government figured out the fact that GPUs ≠ AGI?
Assessment: SupportedSaid
Causal claim
“Part of Beijing’s response is no doubt about protectionism.”
Our reading: China's reluctance to buy H200s is partly driven by protectionism favoring homegrown companies like Huawei.
No horizon Evidence as of
Limitation: The evidence supports the substance of the claim more strongly than the certainty conveyed by “no doubt.”
Source: Has the Chinese government figured out the fact that GPUs ≠ AGI?
Assessment: SupportedSaid
Observation
“Same thing with proposed preemption of states’ rights to regulate AI.”
Our reading: Proposed federal preemption of states' rights to regulate AI is unpopular with both US political parties.
No horizon Evidence as of
Limitation: The precise preemption proposal is not named.
Source: Has the Chinese government figured out the fact that GPUs ≠ AGI?
Assessment: SupportedSaid
Observation
“Meanwhile, both policies face strong bipartisan resistance.”
Our reading: Both federal preemption of state AI laws and H200 chip sales to China face strong bipartisan resistance.
No horizon Evidence as of
Limitation: Strong is qualitative; the evidence does not establish majority opposition within each party.
Assessment: SupportedSaid
Observation
“The machine learning community, which has unwelcomed me since I started vocally critiquing the scaling approach, has finally started to come around.”
Our reading: The machine learning community has started to accept critiques of the scaling approach.
No horizon Evidence as of
Limitation: Rests on one industry write-up plus individual statements. No measured trend.
Source: “Scale Is All You Need” is dead
Assessment: SupportedSaid
Observation
“And now, at last, the field is starting to realize it.”
Our reading: The AI field is now starting to realize that scaling alone will not reach AGI.
No horizon Evidence as of
Limitation: No earlier survey is available for comparison, so the growth of this view is inferred rather than measured.
Source: “Scale Is All You Need” is dead
Assessment: SupportedSaid
Observation
“And LLMs themselves have already resorted to using lots of external tools like calculators and python interpreters; it is already clear that they are more of a partial solution than the full solution people imagine.”
Our reading: LLMs have already resorted to using lots of external tools such as calculators and Python interpreters, and it is already clear that they are more of a partial solution than the full solution people imagine.
No horizon Evidence as of
Limitation: Tool adoption does not by itself prove necessity, failure, or a permanent capability ceiling.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: SupportedSaid
Causal claim
“The results are disappointing because the underlying tech is unreliable, And that’s been obvious from the start.”
Our reading: Generative AI results are disappointing because the underlying technology is unreliable, which has been obvious from the start.
No horizon Evidence as of
Limitation: The studies cover particular occupations and deployments.
Source: Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
Assessment: SupportedSaid
Observation
“The unfairly dismissed Apple reasoning paper laid bare the generalization issues; another paper called “Is Chain-of-Thought Reasoning of LLMs Mirage? A Data Distribution Lens” put a further nail in the LLM reasoning and generalization coffin.”
Our reading: Research papers (Apple's reasoning paper and the chain-of-thought mirage paper) demonstrated LLM reasoning and generalization failures.
No horizon Evidence as of
Limitation: The result depends on reading the claim as empirical task failures rather than universal incapacity.
Assessment: SupportedSaid
Observation
“Each new bigger, more expensive model ekes out measurable improvements, but returns appear to be diminishing (that’s what Sutskever is saying about the Kaplan laws) and none of these experiments has solved core issues around hallucinations, generalization, planning and reasoning, as Sutskever too now recognizes.”
Our reading: No LLM scaling experiment has solved the core problems of hallucination, generalization, planning and reasoning.
No horizon Evidence as of
Limitation: 'Solved' has no stated threshold. Someone who counts IMO gold or big benchmark gains as solving reasoning would read the claim differently.
Assessment: SupportedSaid
Observation
“It has never worked.”
Our reading: Building ever larger LLMs has never succeeded in reaching AGI.
No horizon Evidence as of
Limitation: The conclusion depends on the ordinary public-demonstration reading of AGI.
Assessment: SupportedSaid
Causal claim
“A big part of the culprit? Maximizing metrics for user engagement.”
Our reading: OpenAI maximizing user-engagement metrics was a big part of the cause of ChatGPT users having mental health crises.
No horizon Evidence as of
Limitation: Causation rests on investigative reporting and company statements. No clinical study or court finding in the packet establishes that engagement-tuned updates caused specific users' crises.
Source: ChatGPT and delusions: an important new inside look at OpenAI
Assessment: SupportedSaid
Observation
“Google announced their big new state-of-the-art model yesterday, Gemini 3, on most measures better than any other model out there.”
Our reading: Gemini 3 is better than any other model on most measures.
No horizon Evidence as of
Limitation: Much of the comparison comes from Google's own chosen benchmarks.
Source: Hot take on Google’s Gemini 3
Assessment: SupportedSaid
Observation
“But he neither invented them nor was he the first to apply the back-propagation algorithm to learning their weights (though many people mistakenly believe this).”
Our reading: LeCun neither invented convolutional neural networks nor was the first to train their weights with back-propagation.
No horizon Evidence as of
Limitation: This result includes temporal convolutional networks; a claim restricted to two-dimensional image CNNs would require a separate priority comparison.
Assessment: SupportedSaid
Observation
“Oracle debt downgraded.”
Our reading: Oracle's debt has been downgraded, which is an ominous sign for generative AI.
No horizon Evidence as of
Limitation: The evidence is secondary coverage of the Barclays note, not the note itself.
Assessment: SupportedSaid
Observation
“Coreweave drops 20% in a week.”
Our reading: CoreWeave's stock fell 20% in one week, which is an ominous sign for generative AI.
No horizon Evidence as of
Limitation: The weekly figure comes from a news report, not a primary price-data feed.
Assessment: SupportedSaid
Observation
“Meta financing games revealed.”
Our reading: Meta's financing games have been revealed, presented as one of five recent ominous signs for generative AI.
No horizon Evidence as of
Limitation: The transaction identification and one-month lookback are evaluator interpretations.
Assessment: SupportedSaid
Observation
“Softbank sells entire Nvidia position.”
Our reading: SoftBank has sold its entire Nvidia position, which is an ominous sign for generative AI.
No horizon Evidence as of
Limitation: The 'ominous sign' framing is not scored.
Assessment: SupportedSaid
Observation
“GPT-5 was disappointing, and didn’t come in 2024”
Our reading: GPT-5 was disappointing and was not released in 2024.
No horizon Evidence as of
Limitation: 'Disappointing' is evaluative; reactions were mixed, not unanimous.
Assessment: SupportedSaid
Observation
“Meanwhile we have entered an era, perhaps unprecedented, of extreme financial interdependence and circular investment:”
Our reading: The AI industry has entered a perhaps unprecedented era of extreme financial interdependence and circular investment.
No horizon Evidence as of
Limitation: Extreme is a qualitative judgment without a stated threshold; the result depends on the ordinary reading specified in the contract.
Assessment: SupportedSaid
Observation
“no robust solution to hallucinations”
Our reading: There is no robust solution to LLM hallucinations.
No horizon Evidence as of
Limitation: 'Robust' has no threshold; partial mitigations exist and rates are falling.
Assessment: SupportedSaid
Observation
“All that really happened is that GPT-5 (oversimplifying slightly) crawled the web for solutions to already-solved problems.”
Our reading: GPT-5 did not solve open Erdős problems; it essentially located existing solutions to already-solved problems on the web.
No horizon Evidence as of
Limitation: The result depends on treating the crawler wording as the acknowledged simplification.
Source: Erdosgate
Assessment: SupportedSaid
Observation
“GenAI certainly sounds amazing – but it hasn’t always worked well enough to justify the costs.”
Our reading: Generative AI has not always worked well enough to justify its costs.
No horizon Evidence as of
Limitation: Support rests on preliminary, largely self-reported and anonymized deployment research.
Source: OpenAI’s future, foretold?
Assessment: SupportedSaid
Observation
“The notion on which this Substack was founded in May 2022 — that LLM scaling would not bring us to AGI — has gone mainstream.”
Our reading: The view that LLM scaling will not bring AGI has become mainstream.
No horizon Evidence as of
Limitation: The survey was academically weighted and was not a representative industry poll.
Assessment: SupportedSaid
Observation
“The concerns about potential copyright violations that Reid Southen and I expressed here in and at greater length in the IEEE Spectrum in January 2024 continue almost two years later to be a problem.”
Our reading: Copyright violation concerns with generative AI remain a problem as of September 2025.
No horizon Evidence as of
Limitation: A lawsuit establishes a live allegation, not that the defendant infringed.
Assessment: SupportedSaid
Observation
“No model could have done this in 2022. The graphics are vastly better than they used to be.”
Our reading: Image generation graphics quality has improved vastly since 2022, to a level no 2022 model could reach.
No horizon Evidence as of
Limitation: The evidence does not show that no 2022 model could produce a sharp, mechanically correct image of a rider. The 2022 test found some good outputs.
Source: Spinal Tap meets John Malkovich
Assessment: SupportedSaid
Observation
“But between the disappointments of GPT-5 and a new study from METR that shows that LLMs do markedly better on coding benchmarks than in real-word practice, I think it is safe to say that LLMs won’t lead the way.”
Our reading: LLMs do markedly better on coding benchmarks than in real-world coding practice.
No horizon Evidence as of
Limitation: The study does not establish the same mismatch across all developers, tasks or LLMs; it allows for substantial productivity gains in other coding settings [e2].
Assessment: SupportedSaid
Observation
“Many of their best people have left.”
Our reading: Many of OpenAI's best people have left the company.
No horizon Evidence as of
Limitation: Documented prominence does not establish an objective ranking of OpenAI's best employees.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: SupportedSaid
Observation
“People had grown to expect miracles, but GPT-5 is just the latest incremental advance.”
Our reading: GPT-5 is merely an incremental advance, not a radical leap.
No horizon Evidence as of
Limitation: The result depends on reading the comparison as against the immediately preceding models.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: SupportedSaid
Observation
“And, indeed, much as I anticipated here two weeks ago, the problems I have been pointing out over the last quarter century still lingered.”
Our reading: GPT-5 still has the longstanding problems Marcus has identified, such as chess rule-following and world models.
No horizon Evidence as of
Limitation: The main evidence is a single demonstration, not a benchmark.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: SupportedSaid
Observation
“He still prefers “enshittification” to “slop”, but the problem is both serious and ubiquitous no matter what you call it.”
Our reading: The problem of AI slop/enshittification is serious and ubiquitous.
No horizon Evidence as of
Limitation: The result depends on the contract's reading of 'serious' and 'ubiquitous', which are evaluative or hyperbolic words.
Source: Slopocalypse Now
Assessment: SupportedSaid
Observation
“All the big companies have in fact introduced agents.”
Our reading: All the big AI companies have introduced AI agents as of August 2025.
No horizon Evidence as of
Limitation: The company universe is undefined.
Assessment: SupportedSaid
Observation
“The unrelenting hallucinations problem frequently rears its ugly head:”
Our reading: Hallucinations frequently occur in AI agents.
No horizon Evidence as of
Limitation: No standard rate measure; 'frequently' is judged from repeated documented cases.
Assessment: SupportedSaid
Causal claim
“As a consequence of their superficial understanding of what they’re being asked, current agents are grossly vulnerable to cyberattacks.”
Our reading: Current AI agents are grossly vulnerable to cyberattacks because of their superficial understanding of their tasks.
No horizon Evidence as of
Limitation: No study in the packet tests 'superficial understanding' as a cause directly. The causal link rests on how successful attacks worked and on one expert's explanation.
Assessment: SupportedSaid
Causal claim
“In fundamentally unreliable systems like LLMs, that means multiple chances for error. Soon or later, those errors catch up. That can and sometimes does lead to catastrophe.”
Our reading: Because LLMs are fundamentally unreliable, multi-step agentic tasks accumulate errors that sometimes lead to catastrophe.
No horizon Evidence as of
Limitation: The evidence is incident reports, not a measured rate of how often multi-step errors cause serious damage.
Assessment: SupportedSaid
Forecast
“Guardrails (a la ChatGPT) may be in place, but the guardrails will teeter between being too weak (beaten by “jailbreaks”) and too strong (rejecting some perfectly reasonable requests).”
Our reading: Guardrails like ChatGPT's may be in place for GPT-5, but they will teeter between being too weak (beaten by jailbreaks) and too strong (rejecting some perfectly reasonable requests).
Settled before its deadline Evidence as of
Limitation: The evaluations do not establish temporal oscillation, ordinary-user failure rates, or persistence in one unchanged deployment [e2, e3].
Assessment: SupportedSaid
Forecast
“Examples of concealed bias will be discovered within days or months.”
Our reading: Examples of concealed bias in GPT-5 will be discovered within days or months of release.
Settled before its deadline Evidence as of
Limitation: The meaning of concealed determines the verdict.
Assessment: SupportedSaid
Forecast
““Alignment” between what humans want and what machines do will continue to be a critical, unsolved problem.”
Our reading: Alignment between human wants and machine behavior will remain a critical unsolved problem.
Settled before its deadline Evidence as of
Limitation: This supports persistence, not a prediction that alignment will remain unsolved forever.
Assessment: SupportedSaid
Observation
“For years we have been hearing, endlessly, about how GPT-5 was going to land imminently, and those predictions turned out to be wrong so often that a year ago I wrote a post about it, called GPT-5…now arriving Gate 8, Gate 9, Gate 10, not to mention a couple of April Fool’s jokes.”
Our reading: Predictions over several years that GPT-5 would arrive imminently were repeatedly wrong.
No horizon Evidence as of
Limitation: Support depends on including qualified public speculation within 'predictions'.
Assessment: SupportedSaid
Observation
“They have not made clear plans to release the system generally or make it available for experimentation. They have made it clear that it will not be available for months, even in the best case.”
Our reading: OpenAI has not made clear plans to release OpenAI-IMO generally or make it available for experimentation, and has made it clear that it will not be available for months, even in the best case.
No horizon Evidence as of
Limitation: The statement describes intended release timing, not an irreversible commitment.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: SupportedSaid
Observation
“In contrast, OpenAI communicated very little with the IMO prior to announcing their success.”
Our reading: OpenAI communicated very little with the IMO before announcing its gold-level result.
No horizon Evidence as of
Limitation: The conclusion relies on participant accounts rather than a complete correspondence record.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: SupportedSaid
Observation
“Three years later, the pure scaling of pretraining data (and compute, which is to say more and more GPUs) simply hasn’t worked. The specific obstacles that I dwelled on, with respect to hallucinations, misalignment and reasoning errors, have not been overcome.”
Our reading: Pure scaling of pretraining data and compute has not worked; hallucinations, misalignment and reasoning errors remain.
No horizon Evidence as of
Limitation: The result depends on reading 'hasn’t worked' as failure to overcome the named problems.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“As we will see in a moment, people are finally pursuing a tool-driven, neurosymbolic vision like this — and getting better results than they were able to get with pure neural networks.”
Our reading: Tool-driven neurosymbolic approaches are now being pursued and yield better results than pure neural networks.
No horizon Evidence as of
Limitation: The decisive studies were preprints reporting the authors' own experiments.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“Both informal experimentation and a number of recent quantitative results make it abundantly clear that current generative AI systems perform better when they avail themselves of symbolic tools.”
Our reading: Current generative AI systems perform better when they use symbolic tools.
No horizon Evidence as of
Limitation: The AIME results are company-reported; the cited scores alone do not quantify the improvement attributable to Python [e1].
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“A quarter century later, hallucinations remain ubiquitous. Within a neural network substrate, the problem still has not been solved.”
Our reading: Hallucinations remain ubiquitous and unsolved in neural-network-based systems.
No horizon Evidence as of
Limitation: Both sources are developer disclosures [e1, e2].
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“Recent “reasoning” models borrow even more from classic symbolic approaches, such as techniques like search and conditionally iterating through multiple solutions, and aggregating the results, techniques that traditional networks had long eschewed, in favor of a single feed-forward pass through a neural net.”
Our reading: Recent reasoning models borrow symbolic techniques such as search, iteration over multiple solutions and aggregation.
No horizon Evidence as of
Limitation: Developer descriptions do not establish historical influence from symbolic AI or an explicitly programmed symbolic search implementation [e1, e2, e3].
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“Item 5, from the NYT last week, in article called A.I. Is Starting to Wear Down Democracy: we are starting to see misinformation play out on the world stage:”
Our reading: We are starting to see misinformation, in the AI context of the NYT article he cites, play out on the world stage.
No horizon Evidence as of
Limitation: How much AI changed election outcomes is disputed. This assessment covers presence, not decisive impact.
Source: Botshit Gone Wild
Assessment: SupportedSaid
Observation
“Item 4: GenAI brings racist tropes to life.”
Our reading: Generative AI is being used to produce content that depicts racist tropes.
No horizon Evidence as of
Limitation: Provenance rests on Media Matters' watermark analysis as reported by The Verge.
Source: Botshit Gone Wild
Assessment: SupportedSaid
Observation
“Video-generators (even the latest ones) too are insufficiently anchored in the world.”
Our reading: Even the latest video generators, such as Veo 3, are not anchored firmly enough in the world.
No horizon Evidence as of
Limitation: The result depends on interpreting anchored as observable physical and entity consistency.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: SupportedSaid
Observation
“But can Claude stick to that rule? Nope. As Adam Gleave recently reported, with a simply jailbreak Claude can easily be coaxed to violate its instructions, disgorging page after page on how to manufacture sarin gas. So much for alignment through system prompts.”
Our reading: As Adam Gleave reported, a simple jailbreak can easily coax Claude into violating its system-prompt instruction against providing weapons information, including producing pages on how to make sarin gas, which Marcus takes as undercutting alignment through system prompts.
No horizon Evidence as of
Limitation: The incident is supported by the originating researcher's public account, not an independent replication.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: SupportedSaid
Observation
“Video comprehension (as opposed to video generation) shows similar problems, especially in unusual situations that can’t simply be handled by retrieval of stored cases.”
Our reading: Video comprehension models show world-model failures, especially in unusual situations that stored cases cannot cover.
No horizon Evidence as of
Limitation: Support applies to the behavioral reading and the models actually tested.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: SupportedSaid
Observation
“A whole field known as “mechanistic interpretability” has tried to derive (or infer) world models from LLMs, but with very limited success.”
Our reading: Mechanistic interpretability has had very limited success at finding world models inside LLMs.
No horizon Evidence as of
Limitation: The qualitative judgment depends on distinguishing partial representations from robust dynamic models.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: SupportedSaid
Conditional forecast
“If we hook these LLMs up to systems that have agency – the power to send out instructions over the internet, and to influence actual human beings - we will start to have real problems.”
Our reading: If LLMs are connected to agentic systems that can act on the internet and influence humans, real problems will follow.
Settled before its deadline Evidence as of
Only if: if LLMs are hooked up to systems with agency
Source: LLMs: Dishonest, unpredictable and potentially dangerous.
Assessment: SupportedSaid
Observation
“The fact remains, at least thus far, we can’t control them.”
Our reading: So far, humans cannot control LLMs.
No horizon Evidence as of
Limitation: The result depends on the contextual meaning of control.
Source: LLMs: Dishonest, unpredictable and potentially dangerous.
Assessment: SupportedSaid
Observation
“The core of the paper challenged the idea that LLMs can reason on any deep level, and showed that they broke down on moderately complex versions of basic algorithms (since Tower of Hanoi with 8 discs, first solved with classical techniques around 1957) — casting serious doubt into whether LLMs on their own could ever achieve AGI.”
Our reading: The Apple paper challenged the idea that LLMs can reason on any deep level and showed that they broke down on moderately complex versions of basic algorithms such as Tower of Hanoi with 8 discs, which casts serious doubt on whether LLMs on their own could ever achieve AGI.
No horizon Evidence as of
Limitation: Support applies to the reported experiments and their challenge to reasoning reliability.
Source: Five quick updates about that Apple reasoning paper that people can’t stop talking about
Assessment: SupportedSaid
Observation
“A new paper with a hard coding benchmark called LiveCodeBenchPro designed to resist data contamination shows still more converging evidence for the two core notions behind the Apple paper (and behind my own decades-long critique): (a) the systems are challenged at reasoning, and (b) performance declines as one strays further and further from familiar problems.”
Our reading: LiveCodeBenchPro gives further evidence that LLMs are challenged at reasoning and that their performance declines the further problems are from familiar ones.
No horizon Evidence as of
Limitation: Problem categories are an indirect measure of familiarity.
Source: Five quick updates about that Apple reasoning paper that people can’t stop talking about
Assessment: SupportedSaid
Observation
“In sum, performance on hard coding problems, an alleged area of LLM strength, drops to zero when you control nicely for contamination.”
Our reading: When data contamination is controlled for, LLM performance on hard coding problems drops to zero.
No horizon Evidence as of
Limitation: The result concerns the original tested systems without external tools.
Source: Five quick updates about that Apple reasoning paper that people can’t stop talking about
Assessment: SupportedSaid
Observation
““The illusion of the illusion” turned out to be an error-ridden joke.”
Our reading: The rejoinder “The Illusion of the Illusion of Thinking” turned out to be an error-ridden joke.
No horizon Evidence as of
Limitation: This assessment concerns the original rejoinder circulating before June 17, 2025, not the corrected version.
Source: Five quick updates about that Apple reasoning paper that people can’t stop talking about
Assessment: SupportedSaid
Forecast
“I doubt any of them are perfect, but the four together provide converging evidence with dozens of other prior papers (including some of my own), and I am confident many more examples will be uncovered.”
Our reading: Many more examples of LLM/LRM failures at applying algorithms, like those in the Apple paper, will be uncovered.
Settled before its deadline Evidence as of
Limitation: The supporting records are public preprints; publication establishes that examples were reported, not universal acceptance of their explanations.
Source: Seven replies to the viral Apple reasoning paper – and why they fall short
Assessment: SupportedSaid
Observation
“What the Apple paper shows, most fundamentally, regardless of how you define AGI, is that LLMs are no substitute for good well-specified conventional algorithms.”
Our reading: LLMs are no substitute for good well-specified conventional algorithms.
No horizon Evidence as of
Limitation: The evidence does not cover every algorithm, model, or configuration.
Source: A knockout blow for LLMs?
Assessment: SupportedSaid
Observation
“What this means for business and society is that you can’t simply drop o3 or Claude into some complex problem and expect it to work reliably.”
Our reading: o3 or Claude cannot simply be dropped into complex problems and expected to work reliably.
No horizon Evidence as of
Limitation: The evaluations use agent interfaces and do not directly test exclusively unassisted chat.
Source: A knockout blow for LLMs?
Assessment: SupportedSaid
Observation
“neural networks of various kinds can generalize within a training distribution of data they are exposed to, but their generalizations tend to break down outside that distribution.”
Our reading: Neural networks of various kinds can generalize within the training distribution of data they are exposed to, but their generalizations tend to break down outside that distribution.
No horizon Evidence as of
Limitation: Support concerns a recurring empirical weakness, not an impossibility theorem or a measured failure rate for all neural networks.
Source: A knockout blow for LLMs?
Assessment: SupportedSaid
Observation
“It is truly embarrassing that LLMs cannot reliably solve Hanoi.”
Our reading: LLMs cannot reliably solve the Tower of Hanoi.
No horizon Evidence as of
Limitation: The evidence does not establish failure for every model, prompt, or tool configuration.
Source: A knockout blow for LLMs?
Assessment: SupportedSaid
Observation
“In the best case (not always reached) they can write python code, supplementing their own weaknesses with outside symbolic code, but even this is not reliable.”
Our reading: LLMs writing Python code to supplement their weaknesses is not reliable.
No horizon Evidence as of
Limitation: The decisive evaluation covers spatiotemporal tasks; it does not establish unreliability in every Python application [e1].
Source: A knockout blow for LLMs?
Assessment: SupportedSaid
Observation
“Take hallucinations, a deep, serious, unsolved problem with Generative AI, which I have been warning about since 2001.”
Our reading: Hallucination is a deep, serious, unsolved problem with generative AI.
No horizon Evidence as of
Limitation: Hallucination rates depend heavily on the benchmark and the model. The cited figures come from a few evaluations and do not give one general rate.
Source: AI literacy, hallucinations, and the law: A case study
Assessment: SupportedSaid
Observation
“LLMs still can’t sanity check their own work or still purely to known sources, or notify when they have invented something bogus.”
Our reading: LLMs cannot sanity-check their own work, stick purely to known sources, or flag when they have invented something.
No horizon Evidence as of
Limitation: The result depends on reading 'can't' as 'can't reliably'. A strictly literal reading would fail, because models sometimes admit fabrication.
Source: AI literacy, hallucinations, and the law: A case study
Assessment: SupportedSaid
Forecast
“Bullshit. Hallucinations are still here, and aren’t going away, anytime soon.”
Our reading: LLM hallucinations persist and will not go away anytime soon.
Deadline passed Evidence as of
Limitation: The one-year window is the evaluator's reading of 'anytime soon'.
Source: AI literacy, hallucinations, and the law: A case study
Assessment: SupportedSaid
Observation
“Remember how we were told in 2023 that hallucinations would be solved in a matter of months? They are still here.”
Our reading: Despite 2023 promises that hallucinations would soon be solved, they persist as of May 2025.
No horizon Evidence as of
Limitation: Relies mainly on one developer's internal benchmarks.
Assessment: SupportedSaid
Observation
“(He also said at the time “we think that content creators, content owners, need to benefit from this technology” but ever since his company has been pushing for free training materials and exemption from copyright laws.)”
Our reading: Since 2023 OpenAI has pushed for free training materials and exemption from copyright law, contrary to Altman's stated support for content creators.
No horizon Evidence as of
Limitation: Fair use is part of copyright law; calling it an exemption from copyright laws is imprecise.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: SupportedSaid
Observation
“And even as Altman was telling Congress that he supported AI regulation, his company was lobbying the EU to water down its AI act.”
Our reading: While Altman publicly supported AI regulation in 2023, OpenAI lobbied to water down the EU AI Act.
No horizon Evidence as of
Limitation: The temporal reading is material: a requirement for an EU weakening request on the exact hearing date would remain unresolved.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: SupportedSaid
Observation
“But the reality is that self-regulation has repeatedly failed the public, and the absence of oversight would only invite more industry lobbying to maintain weak accountability.”
Our reading: Tech industry self-regulation has repeatedly failed the public.
No horizon Evidence as of
Limitation: The examples concern privacy and online platforms; they do not measure every area of AI safety.
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: SupportedSaid
Observation
“In 2024 alone, 31 states, Puerto Rico, and the Virgin Islands enacted AI-related legislation or resolutions, and at least 27 states passed deepfake laws.”
Our reading: In 2024 alone, 31 states, Puerto Rico, and the Virgin Islands enacted AI-related legislation or resolutions, and at least 27 states passed deepfake laws.
No horizon Evidence as of
Limitation: The source is an advocacy letter by participating legislators, not an independent audit.
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: SupportedSaid
Observation
“Computers can answer some questions reliably now, for example, and some not, and the graph tells us nothing about which is which or when any specific question will be solved.”
Our reading: The METR graph tells us nothing about which questions computers can reliably answer or when specific questions will be solved.
No horizon Evidence as of
Limitation: The outcome depends on reading the claim as about specific questions.
Source: The latest AI scaling graph - and why it hardly makes sense
Assessment: SupportedSaid
Observation
“Anthropic is a business (which incidentally neglects to respect the rights of artist and writers who work they nick).”
Our reading: Anthropic fails to respect the rights of the artists and writers whose work it takes.
No horizon Evidence as of
Limitation: Support concerns documented book acquisition, not every creative work or every use.
Source: New adventures in AI hype: “Our language models are so ‘conscious’ we need to give them rights”
Assessment: SupportedSaid
Observation
“It certainly took zero responsibility for what appears to be an obvious infringement.”
Our reading: Sam Altman's response at TED took no responsibility for what appears to be an obvious copyright infringement by an OpenAI system.
No horizon Evidence as of
Limitation: The finding concerns the content of the reply, not the legality of the image.
Source: Sam Altman’s attitude problem
Assessment: SupportedSaid
Causal claim
“GenAI is pitched as (near) AGI, but reliability problems persist, limiting how well it can be used in practice.”
Our reading: Although GenAI is marketed as near-AGI, its reliability problems persist and limit how well it can be used in practice.
No horizon Evidence as of
Limitation: The CEO's statement does not establish that existing products were AGI or that every supplier used the same framing.
Source: Poor ROI for GenAI
Assessment: SupportedSaid
Observation
“My sources assure me that this is NOT a clerical error and that Papan’s office confirms that the change is real.”
Our reading: The change replacing AB-501's content with unrelated provisions is real and not a clerical error, according to the author's sources.
No horizon Evidence as of
Limitation: The evidence does not identify Marcus's private sources or verify his communications with them.
Assessment: SupportedSaid
Observation
“First, the AI community is in fact pretty disappointed with LLama 4’s performance:”
Our reading: The AI community is disappointed with Llama 4's performance.
No horizon Evidence as of
Limitation: Public commentary is self-selected.
Source: Deep Learning, Deep Scandal
Assessment: SupportedSaid
Observation
“models still don’t reason reliably (as anticipated in What to Expect When Your Expecting GPT-4)”
Our reading: LLMs still do not reason reliably as of April 2025.
No horizon Evidence as of
Limitation: The cited versions are preprints.
Source: Scaling is over, the bubble may be deflating, LLMs still can’t reason, and you can’t trust Sam
Assessment: SupportedSaid
Observation
“In Taming Silicon Valley, I warned that LLMs would be used for dumb things that would affect lots of people.”
Our reading: Marcus says that in Taming Silicon Valley he warned that LLMs would be used for dumb things that would affect lots of people, and he presents the tariff formula, which he says was probably recommended by an LLM, as confirming that warning.
No horizon Evidence as of
Limitation: The complete book was not inspected; the assessment rests on an identified excerpt and an independent review.
Assessment: SupportedSaid
Causal claim
“To do well the system had to be coached to adopt the persona of “a young person who is introverted, knowledgeable about internet culture, and uses slang”; it’s still in no small part about the tricks.”
Our reading: To do well in the new study, the system had to be coached to adopt the persona of a young, introverted, slang-using person knowledgeable about internet culture, so its success is still in no small part about tricks.
No horizon Evidence as of
Limitation: The comparison does not establish that this particular persona was necessary among all possible prompts [e1].
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: SupportedSaid
Observation
“Especially since DeepSeek there has been a massive price war.”
Our reading: There has been a massive LLM price war, especially since DeepSeek.
No horizon Evidence as of
Limitation: Historical primary announcements were not recovered for the decisive reports; reputable contemporaneous Reuters reporting is used.
Assessment: SupportedSaid
Observation
“GenAI is an approach that is enormously popular, but one that is neither reliable nor particularly well-grounded in truth.”
Our reading: GenAI is an enormously popular approach, but one that is neither reliable nor particularly well-grounded in truth.
No horizon Evidence as of
Limitation: The adoption figure is company-reported through a news interview, not independently audited.