Claims checked against evidence by Gary Marcus, page 5 of 15
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Claims checked against evidence by Gary Marcus (1403)
Each of these claims was checked against dated evidence.
Page 5 of 15: claims 401 to 500 of 1403, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Forecast
“Examples of concealed bias will be discovered within days or months.”
Our reading: Examples of concealed bias in GPT-5 will be discovered within days or months of release.
Settled before its deadline Evidence as of
Limitation: The meaning of concealed determines the verdict.
Assessment: SupportedSaid
Forecast
““Alignment” between what humans want and what machines do will continue to be a critical, unsolved problem.”
Our reading: Alignment between human wants and machine behavior will remain a critical unsolved problem.
Settled before its deadline Evidence as of
Limitation: This supports persistence, not a prediction that alignment will remain unsolved forever.
Assessment: UnresolvedSaid
Forecast
““Scaling” alone—building bigger and models until they absorb the entire internet — will prove useful, but only to a point.”
Our reading: Scaling alone will prove useful only up to a point.
No deadline stated Evidence as of
Limitation: Finite experiments cannot establish what all future increases in scale will achieve.
Assessment: SupportedSaid
Observation
“For years we have been hearing, endlessly, about how GPT-5 was going to land imminently, and those predictions turned out to be wrong so often that a year ago I wrote a post about it, called GPT-5…now arriving Gate 8, Gate 9, Gate 10, not to mention a couple of April Fool’s jokes.”
Our reading: Predictions over several years that GPT-5 would arrive imminently were repeatedly wrong.
No horizon Evidence as of
Limitation: Support depends on including qualified public speculation within 'predictions'.
Assessment: UnresolvedSaid
Observation
“Neither DeepMind nor OpenAI has given any substantial information about how their systems are designed and trained.”
Our reading: Neither DeepMind nor OpenAI has given substantial information on how their IMO systems are designed and trained.
No horizon Evidence as of
Limitation: Substantial is a judgment threshold, not a specified quantity.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: UnresolvedSaid
Forecast
“Eric Schmidt’s recent prediction (made before IMO 2025), that there will be AI-based “world-class mathematicians” within a year, is pure fantasy.”
Our reading: There will not be AI-based world-class mathematicians within a year (by about July 2026).
Deadline passed Evidence as of
Limitation: The evidence supplies no agreed standard connecting one exceptional result to world-class research capability.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: UnresolvedSaid
Forecast
“Tying for 27th place in a highly elite high school mathematics competition will not immediately change that.”
Our reading: Tying for 27th place at the IMO will not immediately change the fact that only one theorem of middling importance (the Robbins conjecture, in 1996) has ever been proved autonomously by an AI system.
No deadline stated Evidence as of
Limitation: “Immediately” has no operational endpoint; open_horizon does not extend it indefinitely.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: ContradictedSaid
Observation
“In the real world, only one theorem of middling importance (the Robbins conjecture) has ever been proved autonomously by an AI system, and that was back in 1996.”
Our reading: Only one theorem of middling importance (the Robbins conjecture, 1996) has ever been proved autonomously by an AI system.
No horizon Evidence as of
Limitation: Mathematical importance is qualitative; the sources do not supply an exact comparison with Robbins.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: SupportedSaid
Observation
“They have not made clear plans to release the system generally or make it available for experimentation. They have made it clear that it will not be available for months, even in the best case.”
Our reading: OpenAI has not made clear plans to release OpenAI-IMO generally or make it available for experimentation, and has made it clear that it will not be available for months, even in the best case.
No horizon Evidence as of
Limitation: The statement describes intended release timing, not an irreversible commitment.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: SupportedSaid
Observation
“In contrast, OpenAI communicated very little with the IMO prior to announcing their success.”
Our reading: OpenAI communicated very little with the IMO before announcing its gold-level result.
No horizon Evidence as of
Limitation: The conclusion relies on participant accounts rather than a complete correspondence record.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: UnresolvedSaid
Observation
“The system is so bound to Musk it even asked his opinion on pizza.”
Our reading: Grok is so bound to Musk that it sometimes searches for Musk's opinion before answering, and in one case even asked his opinion on pizza.
No horizon Evidence as of
Limitation: The available demonstration establishes occurrence, not how often the behavior happened or its underlying cause.
Assessment: UnresolvedSaid
Observation
“From this I infer that xAI’s main methodology for handling alignment is trial-and-error.”
Our reading: From xAI's account of the incident, Marcus infers that xAI's main methodology for handling alignment is trial-and-error.
No horizon Evidence as of
Limitation: Public disclosures may omit important internal practices.
Assessment: UnresolvedSaid
Observation
“My p(doom) is still lower than most industry people’s.”
Our reading: Marcus's p(doom) is lower than that of most AI industry people.
No horizon Evidence as of
Limitation: A researcher sample is not automatically representative of the AI industry.
Assessment: UnresolvedSaid
Observation
“In contrast, o3, when it draws on symbolic code (which it sometimes displays explicitly) can build crossword grids like this far faster than I could (short of my writing my own code).”
Our reading: When o3 draws on symbolic code, it can build crossword grids like this one far faster than Marcus could, short of writing his own code.
No horizon Evidence as of
Limitation: The informal experiment may have private records that are not publicly available.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“Three years later, the pure scaling of pretraining data (and compute, which is to say more and more GPUs) simply hasn’t worked. The specific obstacles that I dwelled on, with respect to hallucinations, misalignment and reasoning errors, have not been overcome.”
Our reading: Pure scaling of pretraining data and compute has not worked; hallucinations, misalignment and reasoning errors remain.
No horizon Evidence as of
Limitation: The result depends on reading 'hasn’t worked' as failure to overcome the named problems.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“As we will see in a moment, people are finally pursuing a tool-driven, neurosymbolic vision like this — and getting better results than they were able to get with pure neural networks.”
Our reading: Tool-driven neurosymbolic approaches are now being pursued and yield better results than pure neural networks.
No horizon Evidence as of
Limitation: The decisive studies were preprints reporting the authors' own experiments.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“OpenAI quietly began doing this to some degree in 2023 with a system called “code interpreter”, in which LLMs (themselves neural networks) call (purely symbolic) Python interpreters. This is literally “putting programs inside”; exactly what Hinton said was a huge mistake.”
Our reading: In 2023, OpenAI quietly began, to some degree, 'putting programs inside' its systems with Code Interpreter, in which LLMs call purely symbolic Python interpreters. Marcus says this is exactly what Hinton called a huge mistake.
No horizon Evidence as of
Limitation: The launch and functionality component is supported.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“Both informal experimentation and a number of recent quantitative results make it abundantly clear that current generative AI systems perform better when they avail themselves of symbolic tools.”
Our reading: Current generative AI systems perform better when they use symbolic tools.
No horizon Evidence as of
Limitation: The AIME results are company-reported; the cited scores alone do not quantify the improvement attributable to Python [e1].
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“A quarter century later, hallucinations remain ubiquitous. Within a neural network substrate, the problem still has not been solved.”
Our reading: Hallucinations remain ubiquitous and unsolved in neural-network-based systems.
No horizon Evidence as of
Limitation: Both sources are developer disclosures [e1, e2].
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: SupportedSaid
Observation
“Recent “reasoning” models borrow even more from classic symbolic approaches, such as techniques like search and conditionally iterating through multiple solutions, and aggregating the results, techniques that traditional networks had long eschewed, in favor of a single feed-forward pass through a neural net.”
Our reading: Recent reasoning models borrow symbolic techniques such as search, iteration over multiple solutions and aggregation.
No horizon Evidence as of
Limitation: Developer descriptions do not establish historical influence from symbolic AI or an explicitly programmed symbolic search implementation [e1, e2, e3].
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“I argued that scaling as it was then defined (pretraining data and compute) alone was insufficient to solve challenges with reasoning, misalignment and hallucinations. (Spoiler alert: it still hasn’t.)”
Our reading: Scaling pretraining data and compute alone has not solved reasoning, misalignment and hallucination problems.
No horizon Evidence as of
Limitation: This is an evidence limitation, not evidence that pretraining alone had solved the problems.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“Yet nearly all funding has been aimed simply at making them larger.”
Our reading: Nearly all AI funding has gone to making LLMs larger.
No horizon Evidence as of
Limitation: The selected annual period cannot settle an unspecified cumulative historical claim.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“OpenAI, without any sort of of public acknowledgement whatsoever, has accidentally vindicated neurosymbolic AI.”
Our reading: Without any public acknowledgement whatsoever, OpenAI has accidentally vindicated neurosymbolic AI.
No horizon Evidence as of
Limitation: Company-reported improvements establish limited practical benefits, not the necessity of neurosymbolic AI for general intelligence.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Observation
“A significant fraction of the money that people have paid for Generative AI has been for coding assistance.”
Our reading: A significant fraction of the money people have paid for generative AI has been for coding assistance.
No horizon Evidence as of
Limitation: The evidence supplies no matched coding-payment numerator and total generative AI-payment denominator covering the same period and market.
Source: BREAKING NEWS: AI coding may not be helping as much as you think
Assessment: UnresolvedSaid
Observation
“When I point to the positive uses of Generative AI, I invariably point to how GenAI serves as kind of valuable autocomplete for coding.”
Our reading: When pointing to positive uses of generative AI, Marcus invariably points to how it serves as kind of a valuable autocomplete for coding.
No horizon Evidence as of
Limitation: Search coverage is incomplete and some interviews are edited.
Source: BREAKING NEWS: AI coding may not be helping as much as you think
Assessment: UnresolvedSaid
Observation
“Many of us in the field have started to get emails regularly from people who believe that they have seen signs of consciousness in LLMs.”
Our reading: Many AI researchers have started regularly receiving emails from people who believe they have seen signs of consciousness in LLMs.
No horizon Evidence as of
Limitation: Private correspondence is difficult to verify through public sources.
Assessment: UnresolvedSaid
Observation
“It took me less than an hour to find half a dozen clear counterexamples in which ChatGPT 4o failed with respect to compositionality, evident even to a teenager.”
Our reading: In less than an hour, Marcus found half a dozen clear counterexamples in which ChatGPT 4o failed with respect to compositionality, failures evident even to a teenager.
No horizon Evidence as of
Limitation: The evidence does not establish Marcus's discovery interval, identify all six claimed prompt-output pairs, or verify their model identity and clarity to non-specialists.
Assessment: UnresolvedSaid
Forecast
“I predict that such counterexamples will still be easy to find at the end of 2027 because of limitations inherent in LLMs.”
Our reading: At the end of 2027 it will still be easy to find clear counterexamples where LLM-based generative AI fails at compositionality, because of limitations inherent in LLMs.
Not yet due Evidence as of
Limitation: Existing failures cannot establish that comparable failures will remain easy to find at the deadline.
Assessment: SupportedSaid
Observation
“Item 5, from the NYT last week, in article called A.I. Is Starting to Wear Down Democracy: we are starting to see misinformation play out on the world stage:”
Our reading: We are starting to see misinformation, in the AI context of the NYT article he cites, play out on the world stage.
No horizon Evidence as of
Limitation: How much AI changed election outcomes is disputed. This assessment covers presence, not decisive impact.
Source: Botshit Gone Wild
Assessment: SupportedSaid
Observation
“Item 4: GenAI brings racist tropes to life.”
Our reading: Generative AI is being used to produce content that depicts racist tropes.
No horizon Evidence as of
Limitation: Provenance rests on Media Matters' watermark analysis as reported by The Verge.
Source: Botshit Gone Wild
Assessment: UnresolvedSaid
Conditional forecast
“As the number of propositions and the complexity of what is represented increases (e.g., to include logical quantifiers), I expect encoding too will turn out to be unreliable.”
Our reading: As propositions become more numerous and complex, LLMs' encoding of them will also prove unreliable.
No deadline stated Evidence as of
Only if: as number and complexity of propositions increases
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: SupportedSaid
Observation
“Video-generators (even the latest ones) too are insufficiently anchored in the world.”
Our reading: Even the latest video generators, such as Veo 3, are not anchored firmly enough in the world.
No horizon Evidence as of
Limitation: The result depends on interpreting anchored as observable physical and entity consistency.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Observation
“Game play continued, and I was able to beat the “smartest AI on earth” in four moves.”
Our reading: Grok 3 lost to Marcus in four moves at a tic-tac-toe variant that used y's and z's.
No horizon Evidence as of
Limitation: A public record may exist outside the indexed sources searched.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: SupportedSaid
Observation
“But can Claude stick to that rule? Nope. As Adam Gleave recently reported, with a simply jailbreak Claude can easily be coaxed to violate its instructions, disgorging page after page on how to manufacture sarin gas. So much for alignment through system prompts.”
Our reading: As Adam Gleave reported, a simple jailbreak can easily coax Claude into violating its system-prompt instruction against providing weapons information, including producing pages on how to make sarin gas, which Marcus takes as undercutting alignment through system prompts.
No horizon Evidence as of
Limitation: The incident is supported by the originating researcher's public account, not an independent replication.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: SupportedSaid
Observation
“Video comprehension (as opposed to video generation) shows similar problems, especially in unusual situations that can’t simply be handled by retrieval of stored cases.”
Our reading: Video comprehension models show world-model failures, especially in unusual situations that stored cases cannot cover.
No horizon Evidence as of
Limitation: Support applies to the behavioral reading and the models actually tested.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Causal claim
“LLMs try — to their peril — to live without classical world models.”
Our reading: LLMs try to function without classical world models, and this harms them.
No horizon Evidence as of
Limitation: The cited experiment concerns planning rather than the full range of LLM failures.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Causal claim
“In short ChatGPT can approximate the game of chess, but it can’t play it reliably, precisely because (despite immense relevant evidence) it never induces proper world model of the board and the rules.”
Our reading: ChatGPT cannot play chess reliably because it never induces a proper world model of the board and the rules.
No horizon Evidence as of
Limitation: The available intervention studies concern earlier ChatGPT experiments or different language models.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Causal claim
“The fact that they can get by as far as they can without explicit world models is actually astonishing. But much of what ails them comes from that design choice.”
Our reading: Much of what goes wrong with LLMs comes from the design choice to do without explicit world models.
No horizon Evidence as of
Limitation: Much and the relevant population of failures remain unspecified.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Causal claim
“And that is why they hallucinate, frequently. They wouldn’t need to do so if they actually had reliable access to relevant databases.”
Our reading: LLMs hallucinate often because they rebuild facts stochastically from fragments of text instead of using structured databases.
No horizon Evidence as of
Limitation: The supporting experiment concerns structured workflow generation.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Causal claim
“A huge number of peculiar LLM errors can be understood in this way.”
Our reading: The lack of world models explains a huge number of peculiar LLM errors.
No horizon Evidence as of
Limitation: The claim supplies no threshold for huge number.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: SupportedSaid
Observation
“A whole field known as “mechanistic interpretability” has tried to derive (or infer) world models from LLMs, but with very limited success.”
Our reading: Mechanistic interpretability has had very limited success at finding world models inside LLMs.
No horizon Evidence as of
Limitation: The qualitative judgment depends on distinguishing partial representations from robust dynamic models.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Observation
“Text in general is better now (though not in this image) but watch comprehension of any time other than 10:10 (most frequent in advertisements) remains dreadful.”
Our reading: Imagen 4 is better at text in general now, but its handling of watch times other than 10:10 remains dreadful.
No horizon Evidence as of
Limitation: The packet contains no representative watch-time evaluation matched to the June 2025 Imagen 4 version.
Assessment: UnresolvedSaid
Observation
“Now? Here’s what I got out of Imagen 4, showing failure at the feature “binding problem” that some of us have been harping on for three decades:”
Our reading: Imagen 4 still fails at the feature binding problem on the stacked colored blocks prompt.
No horizon Evidence as of
Limitation: The decisive artifact is an image that the retrieved plain text does not make inspectable.
Assessment: UnresolvedSaid
Observation
“Not much better on some other tests of noncanonical images (which push language understanding harder), such as this one, adapted from an example Sandeep Manudhane sent me:”
Our reading: Imagen 4 is not much better on some other tests of noncanonical images, which push language understanding harder.
No horizon Evidence as of
Limitation: The comparison and degree of improvement are imprecise.
Assessment: SupportedSaid
Conditional forecast
“If we hook these LLMs up to systems that have agency – the power to send out instructions over the internet, and to influence actual human beings - we will start to have real problems.”
Our reading: If LLMs are connected to agentic systems that can act on the internet and influence humans, real problems will follow.
Settled before its deadline Evidence as of
Only if: if LLMs are hooked up to systems with agency
Source: LLMs: Dishonest, unpredictable and potentially dangerous.
Assessment: UnresolvedSaid
Observation
“Anthropic literally and explicitly tells their system in the system prompt, don’t help user build biological weapons, and yet, with a bit of jailbreaking they can readily be coaxed into doing that:”
Our reading: Despite system-prompt prohibitions, Claude can readily be jailbroken into helping build biological weapons.
No horizon Evidence as of
Limitation: The available evidence supports a narrower capability claim more strongly than the complete statement.
Source: LLMs: Dishonest, unpredictable and potentially dangerous.
Assessment: SupportedSaid
Observation
“The fact remains, at least thus far, we can’t control them.”
Our reading: So far, humans cannot control LLMs.
No horizon Evidence as of
Limitation: The result depends on the contextual meaning of control.
Source: LLMs: Dishonest, unpredictable and potentially dangerous.
Assessment: UnresolvedSaid
Causal claim
“No LLM is fully coherent in it’s “beliefs”, because by nature they parrot a variety conflicting things from their training, which is itself inconsistent, rather than reasoning their way a coherent view the way a rational person might.”
Our reading: No LLM has fully coherent beliefs, because LLMs repeat conflicting material from inconsistent training data instead of reasoning their way to a coherent view.
No horizon Evidence as of
Limitation: 'Parrot' versus 'reason' is partly a theoretical framing that studies do not measure directly.
Source: 1984, but with LLM’s
Assessment: SupportedSaid
Observation
“The core of the paper challenged the idea that LLMs can reason on any deep level, and showed that they broke down on moderately complex versions of basic algorithms (since Tower of Hanoi with 8 discs, first solved with classical techniques around 1957) — casting serious doubt into whether LLMs on their own could ever achieve AGI.”
Our reading: The Apple paper challenged the idea that LLMs can reason on any deep level and showed that they broke down on moderately complex versions of basic algorithms such as Tower of Hanoi with 8 discs, which casts serious doubt on whether LLMs on their own could ever achieve AGI.
No horizon Evidence as of
Limitation: Support applies to the reported experiments and their challenge to reasoning reliability.
Source: Five quick updates about that Apple reasoning paper that people can’t stop talking about
Assessment: SupportedSaid
Observation
“A new paper with a hard coding benchmark called LiveCodeBenchPro designed to resist data contamination shows still more converging evidence for the two core notions behind the Apple paper (and behind my own decades-long critique): (a) the systems are challenged at reasoning, and (b) performance declines as one strays further and further from familiar problems.”
Our reading: LiveCodeBenchPro gives further evidence that LLMs are challenged at reasoning and that their performance declines the further problems are from familiar ones.
No horizon Evidence as of
Limitation: Problem categories are an indirect measure of familiarity.
Source: Five quick updates about that Apple reasoning paper that people can’t stop talking about
Assessment: SupportedSaid
Observation
“In sum, performance on hard coding problems, an alleged area of LLM strength, drops to zero when you control nicely for contamination.”
Our reading: When data contamination is controlled for, LLM performance on hard coding problems drops to zero.
No horizon Evidence as of
Limitation: The result concerns the original tested systems without external tools.
Source: Five quick updates about that Apple reasoning paper that people can’t stop talking about
Assessment: SupportedSaid
Observation
““The illusion of the illusion” turned out to be an error-ridden joke.”
Our reading: The rejoinder “The Illusion of the Illusion of Thinking” turned out to be an error-ridden joke.
No horizon Evidence as of
Limitation: This assessment concerns the original rejoinder circulating before June 17, 2025, not the corrected version.
Source: Five quick updates about that Apple reasoning paper that people can’t stop talking about
Assessment: SupportedSaid
Forecast
“I doubt any of them are perfect, but the four together provide converging evidence with dozens of other prior papers (including some of my own), and I am confident many more examples will be uncovered.”
Our reading: Many more examples of LLM/LRM failures at applying algorithms, like those in the Apple paper, will be uncovered.
Settled before its deadline Evidence as of
Limitation: The supporting records are public preprints; publication establishes that examples were reported, not universal acceptance of their explanations.
Source: Seven replies to the viral Apple reasoning paper – and why they fall short
Assessment: UnresolvedSaid
Causal claim
“But downloading code without conceptual understanding is of less help in the case of new problems, dynamically changing environments, and so on.”
Our reading: LLMs retrieving code without conceptual understanding help less on new problems and dynamically changing environments.
No horizon Evidence as of
Limitation: The meaning of conceptual understanding remains unsettled in this claim.
Source: Seven replies to the viral Apple reasoning paper – and why they fall short
Assessment: UnresolvedSaid
Forecast
“Mark my words: in time, we will see scores of papers reinforcing the Apple results.”
Our reading: Over time, scores of papers will reinforce the Apple paper's findings on reasoning model limitations.
No deadline stated Evidence as of
Limitation: The bibliography search is not exhaustive.
Source: Seven replies to the viral Apple reasoning paper – and why they fall short
Assessment: UnresolvedSaid
Observation
“But also, we need to do so reliably, and in a general way and we haven’t yet crossed that threshold.”
Our reading: Current AI does not yet integrate neural and symbolic methods reliably and generally.
No horizon Evidence as of
Limitation: The claim supplies neither a reliability measure nor a required range of tasks.
Source: Seven replies to the viral Apple reasoning paper – and why they fall short
Assessment: SupportedSaid
Observation
“What the Apple paper shows, most fundamentally, regardless of how you define AGI, is that LLMs are no substitute for good well-specified conventional algorithms.”
Our reading: LLMs are no substitute for good well-specified conventional algorithms.
No horizon Evidence as of
Limitation: The evidence does not cover every algorithm, model, or configuration.
Source: A knockout blow for LLMs?
Assessment: UnresolvedSaid
Forecast
“At least for the next decade, LLMs (with and without inference time “reasoning”) will continue have their uses, especially for coding and brainstorming and writing.”
Our reading: For at least the next decade LLMs will continue to have uses, especially for coding, brainstorming and writing.
Not yet due Evidence as of
Limitation: The evidence covers one provider and an interim usage period.
Source: A knockout blow for LLMs?
Assessment: SupportedSaid
Observation
“What this means for business and society is that you can’t simply drop o3 or Claude into some complex problem and expect it to work reliably.”
Our reading: o3 or Claude cannot simply be dropped into complex problems and expected to work reliably.
No horizon Evidence as of
Limitation: The evaluations use agent interfaces and do not directly test exclusively unassisted chat.
Source: A knockout blow for LLMs?
Assessment: UnresolvedSaid
Observation
“Claude, on the other hand, can barely do 7 discs, getting less than 80% accuracy, left bottom panel below, and pretty much can’t get 8 correct at all.”
Our reading: Claude gets under 80% accuracy on 7-disc Tower of Hanoi and essentially cannot solve 8 discs.
No horizon Evidence as of
Limitation: The original PDFs exceeded the reading tool's size limit.
Source: A knockout blow for LLMs?
Assessment: SupportedSaid
Observation
“neural networks of various kinds can generalize within a training distribution of data they are exposed to, but their generalizations tend to break down outside that distribution.”
Our reading: Neural networks of various kinds can generalize within the training distribution of data they are exposed to, but their generalizations tend to break down outside that distribution.
No horizon Evidence as of
Limitation: Support concerns a recurring empirical weakness, not an impossibility theorem or a measured failure rate for all neural networks.
Source: A knockout blow for LLMs?
Assessment: UnresolvedSaid
Observation
“(They also can’t play chess as well as conventional algorithms, can’t fold proteins like special-purpose neurosymbolic hybrids, can’t run databases as well as conventional databases, etc.)”
Our reading: LLMs can't play chess, fold proteins, or run databases as well as conventional or special-purpose systems.
No horizon Evidence as of
Limitation: The packet supplies no qualifying protein-structure or database-execution comparison.
Source: A knockout blow for LLMs?
Assessment: SupportedSaid
Observation
“It is truly embarrassing that LLMs cannot reliably solve Hanoi.”
Our reading: LLMs cannot reliably solve the Tower of Hanoi.
No horizon Evidence as of
Limitation: The evidence does not establish failure for every model, prompt, or tool configuration.
Source: A knockout blow for LLMs?
Assessment: UnresolvedSaid
Forecast
“But anybody who thinks LLMs are a direct route to the sort AGI that could fundamentally transform society for the good is kidding themselves.”
Our reading: Anybody who thinks LLMs are a direct route to the sort of AGI that could fundamentally transform society for the good is kidding themselves.
No deadline stated Evidence as of
Limitation: The result depends on an evaluator-supplied reading of AGI and 'direct route'.
Source: A knockout blow for LLMs?
Assessment: SupportedSaid
Observation
“In the best case (not always reached) they can write python code, supplementing their own weaknesses with outside symbolic code, but even this is not reliable.”
Our reading: LLMs writing Python code to supplement their weaknesses is not reliable.
No horizon Evidence as of
Limitation: The decisive evaluation covers spatiotemporal tasks; it does not establish unreliability in every Python application [e1].
Source: A knockout blow for LLMs?
Assessment: UnresolvedSaid
Observation
“[Aside from this, models like o3 actually hallucinate a bunch more than attentive humans, struggle heavily with drawing reliable diagrams, etc; they happen to share a few weakness with humans, but on a bunch of dimensions they actually fall short.]”
Our reading: Models like o3 hallucinate much more than attentive humans and struggle with reliable diagrams.
No horizon Evidence as of
Limitation: The claim combines two distinct empirical assertions.
Source: A knockout blow for LLMs?
Assessment: SupportedSaid
Observation
“Take hallucinations, a deep, serious, unsolved problem with Generative AI, which I have been warning about since 2001.”
Our reading: Hallucination is a deep, serious, unsolved problem with generative AI.
No horizon Evidence as of
Limitation: Hallucination rates depend heavily on the benchmark and the model. The cited figures come from a few evaluations and do not give one general rate.
Source: AI literacy, hallucinations, and the law: A case study
Assessment: SupportedSaid
Observation
“LLMs still can’t sanity check their own work or still purely to known sources, or notify when they have invented something bogus.”
Our reading: LLMs cannot sanity-check their own work, stick purely to known sources, or flag when they have invented something.
No horizon Evidence as of
Limitation: The result depends on reading 'can't' as 'can't reliably'. A strictly literal reading would fail, because models sometimes admit fabrication.
Source: AI literacy, hallucinations, and the law: A case study
Assessment: UnresolvedSaid
Observation
“In all, Charlotin’s database lists 112 cases, for an average of a bit less than one publicly reported case a week since ChatGPT become popular. And if May is representative, the problem is getting worse.”
Our reading: Charlotin's database lists 112 cases, an average of a bit less than one publicly reported case a week since ChatGPT became popular, and if May is representative, the problem is getting worse.
No horizon Evidence as of
Only if: if May is representative
Source: AI literacy, hallucinations, and the law: A case study
Assessment: SupportedSaid
Forecast
“Bullshit. Hallucinations are still here, and aren’t going away, anytime soon.”
Our reading: LLM hallucinations persist and will not go away anytime soon.
Deadline passed Evidence as of
Limitation: The one-year window is the evaluator's reading of 'anytime soon'.
Source: AI literacy, hallucinations, and the law: A case study
Assessment: UnresolvedSaid
Conditional forecast
“A camera and microphone. You would be monitored all the time. Everything you say, everything you do, and not just recorded, but analyzed, interpreted and turned into data, which of course OpenAI train their models on.”
Our reading: If OpenAI's planned io gadget is, as it would seem, an always-worn device with a camera and microphone, its users would be monitored all the time, with everything they say and do recorded, analyzed, interpreted and turned into data that OpenAI trains its models on.
No deadline stated Evidence as of
Only if: if the gadget is, as it would seem, worn all the time with a camera and microphone
Source: Black Mirror was a warmup act
Assessment: UnresolvedSaid
Observation
“The US has done almost nothing legislatively about the risks of AI (aside from deepfake porn).”
Our reading: The US has passed almost no legislation addressing AI risks, apart from deepfake porn.
No horizon Evidence as of
Limitation: The evidence items do not provide a complete inventory of federal protections enacted through May 22, 2025.
Assessment: UnresolvedSaid
Causal claim
“Their stories about this are, in my view, greatly flawed, but having outside groups with science fiction chops writing stuff like this distracts away from those flaws, and gives more power to the very companies trying hardest to race towards AGI.”
Our reading: OpenAI's and Anthropic's AGI-imminence stories are greatly flawed, and outside scenarios like AI 2027 distract from those flaws and empower those companies.
No horizon Evidence as of
Limitation: The criticism of the company stories is underspecified and partly evaluative.
Assessment: UnresolvedSaid
Observation
“And so far, anyway, the impact on the labor market of Generative AI has been modest; ditto for productivity in any measurable statistics.”
Our reading: So far, generative AI's impact on the labor market and on measurable productivity has been modest.
No horizon Evidence as of
Limitation: Geographic and occupational coverage is limited.
Assessment: UnresolvedSaid
Causal claim
“AI 2027 isn’t slowing them down; it’s putting wind (and money, and political power) in their sails.”
Our reading: AI 2027 is giving AI companies like OpenAI and Anthropic more money and political power rather than slowing them.
No horizon Evidence as of
Limitation: Investor motives and private political discussions may be unavailable.
Assessment: UnresolvedSaid
Forecast
“It’s also encouraging the world to make short-term choices about AI (e.g., making plans around export controls when China will inevitably catch up) rather than longer-term choices (investing in research to develop safer, more alignable approaches to AI).”
Our reading: China will inevitably catch up in AI despite export controls.
No deadline stated Evidence as of
Limitation: The interpretation of catch up materially affects the result.
Assessment: SupportedSaid
Observation
“Remember how we were told in 2023 that hallucinations would be solved in a matter of months? They are still here.”
Our reading: Despite 2023 promises that hallucinations would soon be solved, they persist as of May 2025.
No horizon Evidence as of
Limitation: Relies mainly on one developer's internal benchmarks.
Assessment: UnresolvedSaid
Causal claim
“Tall tales about the imminence of AGI aren’t slowing down the AI race dynamic that the authors of AI-2027 want to mitigate; they are speeding that very dynamic up.”
Our reading: Stories claiming AGI is imminent are speeding up, not slowing, the AI race dynamic.
No horizon Evidence as of
Limitation: The accessible experimental evidence measures attention, not development decisions.
Assessment: UnresolvedSaid
Observation
“Cell phones and the internet were pretty big, but it’s not obvious their impact was greater than the industrial revolution, and so far AI’s impact has (despite all the PR) been far less than either cell phones or the internet.”
Our reading: So far, AI's impact has been far less than that of cell phones or the internet.
No horizon Evidence as of
Limitation: The evidence provides no common measure of accumulated impact or threshold for 'far less'.
Assessment: ContradictedSaid
Observation
“Absolutely nothing is being done about AI-generated misinformation, political or medical.”
Our reading: Nothing is being done about political or medical AI-generated misinformation.
No horizon Evidence as of
Limitation: The measure targets caller authentication and transmission practices, not every form of misinformation.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: ContradictedSaid
Observation
“Now he is doing everything he can to stop AI regulation of any meaningful sort.”
Our reading: Sam Altman is doing everything he can to stop meaningful AI regulation.
No horizon Evidence as of
Limitation: OpenAI's endorsement establishes an institutional position, not a separate personal endorsement by Altman or his private beliefs.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: ContradictedSaid
Observation
“The shift in norms has extended to our government, too; gone are the days when Washington was afraid of the risks of AI enough to seriously consider doing anything about them.”
Our reading: Washington no longer seriously considers acting on AI risks.
No horizon Evidence as of
Limitation: Targeted measures do not establish comprehensive regulation of frontier AI developers.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: MixedSaid
Observation
“Congress has failed to pass any meaningful AI regulation, and even worse, they are now actively aiming to prevent States — probably our last hope — from passing anything meaningful.”
Our reading: Congress has passed no meaningful AI regulation and is trying to prevent states from passing meaningful AI regulation.
No horizon Evidence as of
Limitation: Congressional passage does not establish enactment or that the bill's duties were already in force.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: SupportedSaid
Observation
“(He also said at the time “we think that content creators, content owners, need to benefit from this technology” but ever since his company has been pushing for free training materials and exemption from copyright laws.)”
Our reading: Since 2023 OpenAI has pushed for free training materials and exemption from copyright law, contrary to Altman's stated support for content creators.
No horizon Evidence as of
Limitation: Fair use is part of copyright law; calling it an exemption from copyright laws is imprecise.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: MixedSaid
Observation
“Government officials aren’t worrying out loud about the risks of AI, anymore. They are downplaying them.”
Our reading: US government officials now downplay rather than voice concern about AI risks.
No horizon Evidence as of
Limitation: The Federal Reserve is part of the federal government but does not speak for the administration.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: SupportedSaid
Observation
“And even as Altman was telling Congress that he supported AI regulation, his company was lobbying the EU to water down its AI act.”
Our reading: While Altman publicly supported AI regulation in 2023, OpenAI lobbied to water down the EU AI Act.
No horizon Evidence as of
Limitation: The temporal reading is material: a requirement for an EU weakening request on the exact hearing date would remain unresolved.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: UnresolvedSaid
Observation
“We have no concretely implemented international AI governance, no national AI agency; we are no longer even positioned well to detect and address AI-escalated cybercrime.”
Our reading: As of May 2025, there is no concretely implemented international AI governance and no US national AI agency, and the US is no longer even well positioned to detect and address AI-escalated cybercrime.
No horizon Evidence as of
Limitation: The OECD article is retrospective and carries an authors' views disclaimer.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: UnresolvedSaid
Observation
“By many accounts, AI-fueled scams have exploded, too, and again there is no coherent federal response.”
Our reading: By many accounts, AI-fueled scams have exploded, and there is no coherent US federal response to them.
No horizon Evidence as of
Limitation: Vendor detection data can change with customer coverage and detection methods.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: UnresolvedSaid
Causal claim
“Because both countries are both building essentially the same kind of models with the same kinds of techniques using the same kinds of data, the results from the two nations are essentially converging on the same outcomes.”
Our reading: US and Chinese AI results are converging because both use the same kinds of models, techniques, and data.
No horizon Evidence as of
Limitation: A supported assessment of benchmark convergence alone would drop the claim's causal commitment.
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: SupportedSaid
Observation
“But the reality is that self-regulation has repeatedly failed the public, and the absence of oversight would only invite more industry lobbying to maintain weak accountability.”
Our reading: Tech industry self-regulation has repeatedly failed the public.
No horizon Evidence as of
Limitation: The examples concern privacy and online platforms; they do not measure every area of AI safety.
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Causal claim
“With no regulatory pressure, tech companies would have little incentive to prioritize safety, transparency, or ethical design; any costs to society would be borne by society.”
Our reading: With no regulatory pressure, tech companies would have little incentive to prioritize safety, transparency, or ethical design, and any costs to society would be borne by society.
No horizon Evidence as of
Only if: with no regulatory pressure
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Conditional forecast
“First, if current trends continue, the “race” with China will not end in victory, for either side.”
Our reading: If current trends continue, neither the US nor China will win the AI race.
No deadline stated Evidence as of
Only if: if current trends continue
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Observation
“So-called leaderboards are no longer dominated by any one country.”
Our reading: AI leaderboards are no longer dominated by any single country.
No horizon Evidence as of
Limitation: Several categories from one benchmark operator are not independent measures.
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: SupportedSaid
Observation
“In 2024 alone, 31 states, Puerto Rico, and the Virgin Islands enacted AI-related legislation or resolutions, and at least 27 states passed deepfake laws.”
Our reading: In 2024 alone, 31 states, Puerto Rico, and the Virgin Islands enacted AI-related legislation or resolutions, and at least 27 states passed deepfake laws.
No horizon Evidence as of
Limitation: The source is an advocacy letter by participating legislators, not an independent audit.
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Forecast
“With enough money you can have TAs and smaller sections of say 20 students each, taught by teaching assistants, but with university budgets being slashed, few universities will have enough money to make that work.”
Our reading: With university budgets being cut, few universities will have enough money to replace term papers with small TA-led sections.
No deadline stated Evidence as of
Limitation: US universities are an interpretive population choice.
Source: ”Everyone is cheating their way through college” with GenAI. Who should bear the costs?
Assessment: UnresolvedSaid
Causal claim
“GenAI has nuked all of this.”
Our reading: GenAI has destroyed term papers' role in giving writing practice, prompting reflection and evaluating students at scale.
No horizon Evidence as of
Limitation: The strongest negative experiment examines take-home examinations rather than term papers.
Source: ”Everyone is cheating their way through college” with GenAI. Who should bear the costs?
Assessment: UnresolvedSaid
Causal claim
“The creators and distributors of GenAI, who have brought this huge problem to society?”
Our reading: The creators and distributors of GenAI have brought this huge problem, GenAI undermining college education and evaluation, to society.
No horizon Evidence as of
Limitation: The strongest direct experiment covers five psychology modules at one UK university.
Source: ”Everyone is cheating their way through college” with GenAI. Who should bear the costs?
Assessment: UnresolvedSaid
Causal claim
“(Those systems also can’t reliably stick to the rules of chess despite having them – and millions of games – in their database, a manifestation of the related problem of extracting statistical tendencies without every fully deriving and apprehending the correct abstractions).”
Our reading: LLMs can't reliably stick to the rules of chess despite having the rules and millions of games in their data, and this is a manifestation of the related problem of extracting statistical tendencies without ever fully deriving and grasping the correct abstractions.
No horizon Evidence as of
Limitation: Puzzle results, full-game results, and playing strength are not interchangeable.
Assessment: UnresolvedSaid
Observation
“They don’t have a database of records like any proper business would (which would be a strong basis to solve the problem); and they don’t have what people like Yann LeCun or Judea Pearl or I would call a world model.”
Our reading: LLMs don't have a database of records like any proper business would, and they don't have what people like Yann LeCun, Judea Pearl, or Marcus would call a world model.
No horizon Evidence as of
Limitation: One documented architecture cannot establish a claim about all LLMs.
Assessment: UnresolvedSaid
Forecast
“We will eventually get to something better, but continuing to put all our eggs in the Henrietta’s LLM basket is absurd to the point of being delusional.”
Our reading: AI will eventually get to something better than LLMs.
No deadline stated Evidence as of
Limitation: The result depends on how broadly 'something better' is read.