Unresolved claims, page 3 of 8
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Unresolved claims (795)
The record was checked, and the evidence available by the cutoff does not settle it. Forecasts whose deadline has not arrived are listed apart, as not yet due.
Page 3 of 8: claims 201 to 300 of 795, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: UnresolvedSaid
Ed Zitron Forecast
“I believe that we’re going to see multiple price increases on AI-native companies in the next few months.”
Our reading: Within a few months of 2025-10-20, AI-native companies (products built on top of AI models) will face multiple price increases.
No deadline stated Evidence as of
Limitation: This record does not assess whether model providers changed their prices after October 2025.
Source: This Is How Much Anthropic and Cursor Spend On Amazon Web Services
Assessment: UnresolvedSaid
Gary Marcus Observation
“June, 2025: the Apple reasoning paper confirmed that even with “reasoning”, LLMs still can’t solve distribution shift, the core Achille’s heel in neural networks that I have been writing about for nearly 30 years.”
Our reading: Even with reasoning, LLMs still cannot handle distribution shift, as the June 2025 Apple paper confirmed.
No horizon Evidence as of
Limitation: The competing papers are evidence of a scientific dispute, not equally decisive demonstrations.
Source: Game over. AGI is not imminent, and LLMs are not the royal road to getting there.
Assessment: UnresolvedSaid
Gary Marcus Observation
“We aren’t close to AGI, but we are reshaping our world on the premise that we are.”
Our reading: We are not close to AGI.
No horizon Evidence as of
Limitation: Benchmark failures do not measure time remaining to AGI.
Source: The Grand AGI Delusion
Assessment: UnresolvedSaid
Gary Marcus Observation
“And schoolchildren using ChatGPT to do their homework aren’t enough to cover that.”
Our reading: Schoolchildren using ChatGPT to do their homework are not enough to cover OpenAI's immense costs and cash burn.
No horizon Evidence as of
Limitation: The reporting window and meaning of 'cover that' are not explicit.
Source: OpenAI’s future, foretold?
Assessment: UnresolvedSaid
Gary Marcus Forecast
“that LLM scaling would not bring us to AGI”
Our reading: Scaling LLMs will not produce AGI.
No deadline stated Evidence as of
Limitation: There is no universally accepted AGI test.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Perhaps I also no longer need to warn people that the idea that coding would disappear soon is bullshit.”
Our reading: Human coding will not disappear soon due to AI.
No deadline stated Evidence as of
Limitation: This leaves the forecast open because of its timing, not because the evidence shows human coding disappeared.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Watch the tuners come and go! So much for object permanence.”
Our reading: Grok's generated video failed to keep objects consistent, with the bass tuners appearing and disappearing.
No horizon Evidence as of
Limitation: No evidence items show what the video contains.
Source: Spinal Tap meets John Malkovich
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Agency plus LLMs have already led multiple “vibe coders” to lose databases, with new reports coming in every day or two, stemming simply from the unreliability inherent in LLMs (that Gary has so often harped on).”
Our reading: Agency combined with LLMs has already led multiple vibe coders to lose databases, with new reports every day or two, and these losses stem simply from the unreliability built into LLMs.
No horizon Evidence as of
Limitation: The Cursor incident rests on a user allegation without execution logs or an independent postmortem [e2].
Assessment: UnresolvedSaid
Gary Marcus Forecast
“The bad news is that although this one was stopped, many others won’t be. No one patch would be enough, or even a thousand; there are just too many variations on the overall theme”
Our reading: Many attacks on AI coding tools won't be stopped; not even a thousand patches would suffice given the number of variations.
No deadline stated Evidence as of
Limitation: The evidence concerns researcher demonstrations rather than a census of malicious attacks.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“But between the disappointments of GPT-5 and a new study from METR that shows that LLMs do markedly better on coding benchmarks than in real-word practice, I think it is safe to say that LLMs won’t lead the way.”
Our reading: LLMs will not be the approach that leads the way to AGI.
No deadline stated Evidence as of
Limitation: The first accepted AGI is an interpretation of 'lead the way', not a criterion Marcus explicitly supplied.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Critically, as I argued at the end of June (and going back to 2019) LLMs never induce proper world models, which is why, for example, they still can’t even play chess reliably,1 and continue to make stupid, head-scratching errors with startling regularity.”
Our reading: LLMs never induce proper world models, which causes them to be unable to play chess reliably and to make frequent stupid errors.
No horizon Evidence as of
Limitation: The strongest intervention evidence concerns a specialist chess model.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“ChatGPT gets them wrong because it is munging words together, not genuinely processing an (elementary) model of British PM history.”
Our reading: ChatGPT gets the British prime minister tasks wrong because it is munging words together rather than genuinely processing an elementary model of British PM history.
No horizon Evidence as of
Limitation: Evidence from Llama-2 cannot directly support or contradict a claim about GPT-5's internal processing.
Assessment: UnresolvedSaid
Gary Marcus Observation
“They don’t induce abstract models of the world that they can reliably engage, even when those models are given to them on a silver platter (as with the rules of chess).”
Our reading: LLMs do not induce abstract world models they can reliably use, even when given the rules explicitly.
No horizon Evidence as of
Limitation: RuleArena tests practical regulations rather than chess.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Even with thinking mode activated, the output was filled with errors that could easily have been avoided, with a 5-second Wikipedia search that any sensible human could have performed.”
Our reading: GPT-5 in thinking mode produced a list of Canadian Prime Ministers full of easily avoidable errors.
No horizon Evidence as of
Limitation: The retrieved plain-text version of the post does not supply independently verifiable image provenance.
Assessment: UnresolvedSaid
Gary Marcus Observation
“The real news is a breaking study from Arizona State University that fully vindicates what I have told you for nearly 30 years—and more recently what Apple told you—about the core weakness of LLMs: their inability to generalize broadly.”
Our reading: LLMs' core weakness is their inability to generalize broadly, as confirmed by an ASU study.
No horizon Evidence as of
Limitation: This assesses the August 5 version available when Marcus wrote.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Driverless cars that still are only available in couple percent of the world’s cities.”
Our reading: Driverless cars are available in only a couple percent of the world's cities.
No horizon Evidence as of
Limitation: This does not establish that worldwide availability exceeded his estimate.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Gary Marcus Observation
“What the mainstream media mostly hasn’t told you yet is that a few days later, hardly anybody is buying Altman’s story.”
Our reading: A few days after launch, hardly anybody believes Altman's claims that GPT-5 is like a PhD-level expert.
No horizon Evidence as of
Limitation: The evidence is news reports of individual reactions, not representative data.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Even some of the tech bros are waking up to the reality that “AGI in 2027” was marketing, not reality.”
Our reading: Even some tech bros are waking up to the reality that 'AGI in 2027' was marketing, not reality.
No horizon Evidence as of
Limitation: The observation concerns public attitudes by August 2025, not whether AGI ultimately arrives in 2027.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Gary Marcus Observation
“The challenge of parts and wholes in generative images that Ernest Davis and I discussed here in December fared no better.”
Our reading: GPT-5 shows no improvement on the parts-and-wholes problem in generated images.
No horizon Evidence as of
Limitation: The only systematic test predates GPT-5 and used ChatGPT-4o.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Elon is moving faster. Anthropic and Google and many others are nipping at their heels. Their relationship with Microsoft has frayed.”
Our reading: Elon Musk is moving faster than OpenAI, Anthropic and Google and many others are nipping at OpenAI's heels, and OpenAI's relationship with Microsoft has frayed.
No horizon Evidence as of
Limitation: Release cadence, benchmark rank and rate of capability improvement are different measures.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Gary Marcus Observation
“It’s not like other systems are faring much better.”
Our reading: Other generative AI systems such as Grok are not faring much better than GPT-5.
No horizon Evidence as of
Limitation: The packet contains no verified primary comparison from the August 7–9 launch window.
Source: GPT-5: Overdue, overhyped and underwhelming. And that’s not the worst of it.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“In practice, those mistakes and errors have greatly limited the utility agents like ChatGPT agent.”
Our reading: Mistakes and errors have greatly limited the utility of agents like ChatGPT agent.
No horizon Evidence as of
Limitation: The causal evidence consists of small, selected task demonstrations.
Assessment: UnresolvedSaid
Gary Marcus Observation
“That sentiment that has only grown over the last several months.”
Our reading: The sentiment that AI agents' reality doesn't match the hype has grown from March to August 2025.
No horizon Evidence as of
Limitation: Trust and disappointment with hype are different measures.
Assessment: UnresolvedSaid
Gary Marcus Observation
“But, none, to my knowledge are reliable, except maybe in very narrow uses cases.”
Our reading: To his knowledge, none of the big companies' AI agents are reliable, except maybe in very narrow use cases.
No horizon Evidence as of
Limitation: The July reports do not provide representative, repeated trials or systematic rates of consequential errors and human intervention [e2, e3].
Assessment: UnresolvedSaid
Gary Marcus Forecast
“For that, we will, I predict, need genuinely new architectures that incorporate explicit knowledge and world models at their very core.”
Our reading: Trustworthy AI will require genuinely new architectures with explicit knowledge and world models at their core.
No deadline stated Evidence as of
Limitation: Behavior on a bounded benchmark cannot establish broad trustworthiness.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“GPT-5 will still, like its predecessors, be a bull in a china shop, reckless and hard to control.”
Our reading: GPT-5, like its predecessors, will be reckless and hard to control.
No deadline stated Evidence as of
Limitation: METR's task suite does not measure the prevalence or consequences of unintended behavior across ordinary deployments, and its detection methods may miss reward hacking [e1].
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Without external aids it won’t be able beat Meta’s Cicero in Diplomacy; it won’t be able to drive a car reliably; it won’t be able to reliably guide a robot like Optimus to be anything like as versatile as Rosie the Robot.”
Our reading: Without external aids, GPT-5 will not be able to beat Meta's Cicero at Diplomacy, will not be able to drive a car reliably, and will not be able to reliably guide a robot like Optimus to be anything like as versatile as Rosie the Robot.
No deadline stated Evidence as of
Limitation: This compound claim cannot be supported from evidence about robotics alone.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Its natural language output still won’t be something that one can reliably hook up to downstream programs; it won’t be something, for example, that you can simply and directly hook up to a database or virtual assistant, with predictable results.”
Our reading: GPT-5's natural language output will not be something one can reliably hook up to downstream programs, for example simply and directly to a database or virtual assistant, with predictable results.
No deadline stated Evidence as of
Limitation: Distinguishing natural language from plaintext code materially affects the assessment.
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“Trustworthy, general artificial intelligence, aligned with human values, will come, when it does, from systems that are more structured, with more built-in knowledge, and will incorporate at least some degree of explicit tools for reasoning and planning, as well as explicit knowledge, that are lacking in systems like GPT.”
Our reading: When trustworthy general AI aligned with human values comes, it will come from systems that are more structured, have more built-in knowledge, and incorporate at least some degree of explicit tools for reasoning and planning, as well as explicit knowledge, that systems like GPT lack.
No deadline stated Evidence as of
Only if: when it does come
Assessment: UnresolvedSaid
Gary Marcus Forecast
“GPT-5 will not have reliable models of the things that it talks about that are accessible to external programmers in a way that reliably feeds downstream processes.”
Our reading: GPT-5 will not have reliable models of the things it talks about that external programmers can access in a way that reliably feeds downstream processes.
No deadline stated Evidence as of
Limitation: The meaning of an accessible subject model materially affects what evidence would count.
Assessment: UnresolvedSaid
Gary Marcus Observation
“I was right on the money then.”
Our reading: Marcus's December 2022 predictions about GPT-4 were accurate.
No horizon Evidence as of
Limitation: The assessment reads the retrospective as referring to all seven predictions; its intended scope is not explicit.
Assessment: UnresolvedSaid
Gary Marcus Forecast
““Scaling” alone—building bigger and models until they absorb the entire internet — will prove useful, but only to a point.”
Our reading: Scaling alone will prove useful only up to a point.
No deadline stated Evidence as of
Limitation: Finite experiments cannot establish what all future increases in scale will achieve.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Neither DeepMind nor OpenAI has given any substantial information about how their systems are designed and trained.”
Our reading: Neither DeepMind nor OpenAI has given substantial information on how their IMO systems are designed and trained.
No horizon Evidence as of
Limitation: Substantial is a judgment threshold, not a specified quantity.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Eric Schmidt’s recent prediction (made before IMO 2025), that there will be AI-based “world-class mathematicians” within a year, is pure fantasy.”
Our reading: There will not be AI-based world-class mathematicians within a year (by about July 2026).
Deadline passed Evidence as of
Limitation: The evidence supplies no agreed standard connecting one exceptional result to world-class research capability.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Tying for 27th place in a highly elite high school mathematics competition will not immediately change that.”
Our reading: Tying for 27th place at the IMO will not immediately change the fact that only one theorem of middling importance (the Robbins conjecture, in 1996) has ever been proved autonomously by an AI system.
No deadline stated Evidence as of
Limitation: “Immediately” has no operational endpoint; open_horizon does not extend it indefinitely.
Source: DeepMind and OpenAI achieve IMO Gold. What does it all mean?
Assessment: UnresolvedSaid
Gary Marcus Observation
“The system is so bound to Musk it even asked his opinion on pizza.”
Our reading: Grok is so bound to Musk that it sometimes searches for Musk's opinion before answering, and in one case even asked his opinion on pizza.
No horizon Evidence as of
Limitation: The available demonstration establishes occurrence, not how often the behavior happened or its underlying cause.
Assessment: UnresolvedSaid
Gary Marcus Observation
“From this I infer that xAI’s main methodology for handling alignment is trial-and-error.”
Our reading: From xAI's account of the incident, Marcus infers that xAI's main methodology for handling alignment is trial-and-error.
No horizon Evidence as of
Limitation: Public disclosures may omit important internal practices.
Assessment: UnresolvedSaid
Gary Marcus Observation
“My p(doom) is still lower than most industry people’s.”
Our reading: Marcus's p(doom) is lower than that of most AI industry people.
No horizon Evidence as of
Limitation: A researcher sample is not automatically representative of the AI industry.
Assessment: UnresolvedSaid
Gary Marcus Observation
“In contrast, o3, when it draws on symbolic code (which it sometimes displays explicitly) can build crossword grids like this far faster than I could (short of my writing my own code).”
Our reading: When o3 draws on symbolic code, it can build crossword grids like this one far faster than Marcus could, short of writing his own code.
No horizon Evidence as of
Limitation: The informal experiment may have private records that are not publicly available.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Gary Marcus Observation
“OpenAI quietly began doing this to some degree in 2023 with a system called “code interpreter”, in which LLMs (themselves neural networks) call (purely symbolic) Python interpreters. This is literally “putting programs inside”; exactly what Hinton said was a huge mistake.”
Our reading: In 2023, OpenAI quietly began, to some degree, 'putting programs inside' its systems with Code Interpreter, in which LLMs call purely symbolic Python interpreters. Marcus says this is exactly what Hinton called a huge mistake.
No horizon Evidence as of
Limitation: The launch and functionality component is supported.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Gary Marcus Observation
“I argued that scaling as it was then defined (pretraining data and compute) alone was insufficient to solve challenges with reasoning, misalignment and hallucinations. (Spoiler alert: it still hasn’t.)”
Our reading: Scaling pretraining data and compute alone has not solved reasoning, misalignment and hallucination problems.
No horizon Evidence as of
Limitation: This is an evidence limitation, not evidence that pretraining alone had solved the problems.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Gary Marcus Observation
“Yet nearly all funding has been aimed simply at making them larger.”
Our reading: Nearly all AI funding has gone to making LLMs larger.
No horizon Evidence as of
Limitation: The selected annual period cannot settle an unspecified cumulative historical claim.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Gary Marcus Observation
“OpenAI, without any sort of of public acknowledgement whatsoever, has accidentally vindicated neurosymbolic AI.”
Our reading: Without any public acknowledgement whatsoever, OpenAI has accidentally vindicated neurosymbolic AI.
No horizon Evidence as of
Limitation: Company-reported improvements establish limited practical benefits, not the necessity of neurosymbolic AI for general intelligence.
Source: How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
Assessment: UnresolvedSaid
Gary Marcus Observation
“A significant fraction of the money that people have paid for Generative AI has been for coding assistance.”
Our reading: A significant fraction of the money people have paid for generative AI has been for coding assistance.
No horizon Evidence as of
Limitation: The evidence supplies no matched coding-payment numerator and total generative AI-payment denominator covering the same period and market.
Source: BREAKING NEWS: AI coding may not be helping as much as you think
Assessment: UnresolvedSaid
Gary Marcus Observation
“When I point to the positive uses of Generative AI, I invariably point to how GenAI serves as kind of valuable autocomplete for coding.”
Our reading: When pointing to positive uses of generative AI, Marcus invariably points to how it serves as kind of a valuable autocomplete for coding.
No horizon Evidence as of
Limitation: Search coverage is incomplete and some interviews are edited.
Source: BREAKING NEWS: AI coding may not be helping as much as you think
Assessment: UnresolvedSaid
Gary Marcus Observation
“Many of us in the field have started to get emails regularly from people who believe that they have seen signs of consciousness in LLMs.”
Our reading: Many AI researchers have started regularly receiving emails from people who believe they have seen signs of consciousness in LLMs.
No horizon Evidence as of
Limitation: Private correspondence is difficult to verify through public sources.
Assessment: UnresolvedSaid
Gary Marcus Observation
“It took me less than an hour to find half a dozen clear counterexamples in which ChatGPT 4o failed with respect to compositionality, evident even to a teenager.”
Our reading: In less than an hour, Marcus found half a dozen clear counterexamples in which ChatGPT 4o failed with respect to compositionality, failures evident even to a teenager.
No horizon Evidence as of
Limitation: The evidence does not establish Marcus's discovery interval, identify all six claimed prompt-output pairs, or verify their model identity and clarity to non-specialists.
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“As the number of propositions and the complexity of what is represented increases (e.g., to include logical quantifiers), I expect encoding too will turn out to be unreliable.”
Our reading: As propositions become more numerous and complex, LLMs' encoding of them will also prove unreliable.
No deadline stated Evidence as of
Only if: as number and complexity of propositions increases
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Gary Marcus Observation
“Game play continued, and I was able to beat the “smartest AI on earth” in four moves.”
Our reading: Grok 3 lost to Marcus in four moves at a tic-tac-toe variant that used y's and z's.
No horizon Evidence as of
Limitation: A public record may exist outside the indexed sources searched.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“LLMs try — to their peril — to live without classical world models.”
Our reading: LLMs try to function without classical world models, and this harms them.
No horizon Evidence as of
Limitation: The cited experiment concerns planning rather than the full range of LLM failures.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“In short ChatGPT can approximate the game of chess, but it can’t play it reliably, precisely because (despite immense relevant evidence) it never induces proper world model of the board and the rules.”
Our reading: ChatGPT cannot play chess reliably because it never induces a proper world model of the board and the rules.
No horizon Evidence as of
Limitation: The available intervention studies concern earlier ChatGPT experiments or different language models.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“The fact that they can get by as far as they can without explicit world models is actually astonishing. But much of what ails them comes from that design choice.”
Our reading: Much of what goes wrong with LLMs comes from the design choice to do without explicit world models.
No horizon Evidence as of
Limitation: Much and the relevant population of failures remain unspecified.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“And that is why they hallucinate, frequently. They wouldn’t need to do so if they actually had reliable access to relevant databases.”
Our reading: LLMs hallucinate often because they rebuild facts stochastically from fragments of text instead of using structured databases.
No horizon Evidence as of
Limitation: The supporting experiment concerns structured workflow generation.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“A huge number of peculiar LLM errors can be understood in this way.”
Our reading: The lack of world models explains a huge number of peculiar LLM errors.
No horizon Evidence as of
Limitation: The claim supplies no threshold for huge number.
Source: Generative AI’s crippling and widespread failure to induce robust models of the world
Assessment: UnresolvedSaid
Gary Marcus Observation
“Text in general is better now (though not in this image) but watch comprehension of any time other than 10:10 (most frequent in advertisements) remains dreadful.”
Our reading: Imagen 4 is better at text in general now, but its handling of watch times other than 10:10 remains dreadful.
No horizon Evidence as of
Limitation: The packet contains no representative watch-time evaluation matched to the June 2025 Imagen 4 version.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Now? Here’s what I got out of Imagen 4, showing failure at the feature “binding problem” that some of us have been harping on for three decades:”
Our reading: Imagen 4 still fails at the feature binding problem on the stacked colored blocks prompt.
No horizon Evidence as of
Limitation: The decisive artifact is an image that the retrieved plain text does not make inspectable.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Not much better on some other tests of noncanonical images (which push language understanding harder), such as this one, adapted from an example Sandeep Manudhane sent me:”
Our reading: Imagen 4 is not much better on some other tests of noncanonical images, which push language understanding harder.
No horizon Evidence as of
Limitation: The comparison and degree of improvement are imprecise.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Anthropic literally and explicitly tells their system in the system prompt, don’t help user build biological weapons, and yet, with a bit of jailbreaking they can readily be coaxed into doing that:”
Our reading: Despite system-prompt prohibitions, Claude can readily be jailbroken into helping build biological weapons.
No horizon Evidence as of
Limitation: The available evidence supports a narrower capability claim more strongly than the complete statement.
Source: LLMs: Dishonest, unpredictable and potentially dangerous.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“No LLM is fully coherent in it’s “beliefs”, because by nature they parrot a variety conflicting things from their training, which is itself inconsistent, rather than reasoning their way a coherent view the way a rational person might.”
Our reading: No LLM has fully coherent beliefs, because LLMs repeat conflicting material from inconsistent training data instead of reasoning their way to a coherent view.
No horizon Evidence as of
Limitation: 'Parrot' versus 'reason' is partly a theoretical framing that studies do not measure directly.
Source: 1984, but with LLM’s
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“But downloading code without conceptual understanding is of less help in the case of new problems, dynamically changing environments, and so on.”
Our reading: LLMs retrieving code without conceptual understanding help less on new problems and dynamically changing environments.
No horizon Evidence as of
Limitation: The meaning of conceptual understanding remains unsettled in this claim.
Source: Seven replies to the viral Apple reasoning paper – and why they fall short
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Mark my words: in time, we will see scores of papers reinforcing the Apple results.”
Our reading: Over time, scores of papers will reinforce the Apple paper's findings on reasoning model limitations.
No deadline stated Evidence as of
Limitation: The bibliography search is not exhaustive.
Source: Seven replies to the viral Apple reasoning paper – and why they fall short
Assessment: UnresolvedSaid
Gary Marcus Observation
“But also, we need to do so reliably, and in a general way and we haven’t yet crossed that threshold.”
Our reading: Current AI does not yet integrate neural and symbolic methods reliably and generally.
No horizon Evidence as of
Limitation: The claim supplies neither a reliability measure nor a required range of tasks.
Source: Seven replies to the viral Apple reasoning paper – and why they fall short
Assessment: UnresolvedSaid
Gary Marcus Observation
“Claude, on the other hand, can barely do 7 discs, getting less than 80% accuracy, left bottom panel below, and pretty much can’t get 8 correct at all.”
Our reading: Claude gets under 80% accuracy on 7-disc Tower of Hanoi and essentially cannot solve 8 discs.
No horizon Evidence as of
Limitation: The original PDFs exceeded the reading tool's size limit.
Source: A knockout blow for LLMs?
Assessment: UnresolvedSaid
Gary Marcus Observation
“(They also can’t play chess as well as conventional algorithms, can’t fold proteins like special-purpose neurosymbolic hybrids, can’t run databases as well as conventional databases, etc.)”
Our reading: LLMs can't play chess, fold proteins, or run databases as well as conventional or special-purpose systems.
No horizon Evidence as of
Limitation: The packet supplies no qualifying protein-structure or database-execution comparison.
Source: A knockout blow for LLMs?
Assessment: UnresolvedSaid
Gary Marcus Forecast
“But anybody who thinks LLMs are a direct route to the sort AGI that could fundamentally transform society for the good is kidding themselves.”
Our reading: Anybody who thinks LLMs are a direct route to the sort of AGI that could fundamentally transform society for the good is kidding themselves.
No deadline stated Evidence as of
Limitation: The result depends on an evaluator-supplied reading of AGI and 'direct route'.
Source: A knockout blow for LLMs?
Assessment: UnresolvedSaid
Gary Marcus Observation
“[Aside from this, models like o3 actually hallucinate a bunch more than attentive humans, struggle heavily with drawing reliable diagrams, etc; they happen to share a few weakness with humans, but on a bunch of dimensions they actually fall short.]”
Our reading: Models like o3 hallucinate much more than attentive humans and struggle with reliable diagrams.
No horizon Evidence as of
Limitation: The claim combines two distinct empirical assertions.
Source: A knockout blow for LLMs?
Assessment: UnresolvedSaid
Gary Marcus Observation
“In all, Charlotin’s database lists 112 cases, for an average of a bit less than one publicly reported case a week since ChatGPT become popular. And if May is representative, the problem is getting worse.”
Our reading: Charlotin's database lists 112 cases, an average of a bit less than one publicly reported case a week since ChatGPT became popular, and if May is representative, the problem is getting worse.
No horizon Evidence as of
Only if: if May is representative
Source: AI literacy, hallucinations, and the law: A case study
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“A camera and microphone. You would be monitored all the time. Everything you say, everything you do, and not just recorded, but analyzed, interpreted and turned into data, which of course OpenAI train their models on.”
Our reading: If OpenAI's planned io gadget is, as it would seem, an always-worn device with a camera and microphone, its users would be monitored all the time, with everything they say and do recorded, analyzed, interpreted and turned into data that OpenAI trains its models on.
No deadline stated Evidence as of
Only if: if the gadget is, as it would seem, worn all the time with a camera and microphone
Source: Black Mirror was a warmup act
Assessment: UnresolvedSaid
Gary Marcus Observation
“The US has done almost nothing legislatively about the risks of AI (aside from deepfake porn).”
Our reading: The US has passed almost no legislation addressing AI risks, apart from deepfake porn.
No horizon Evidence as of
Limitation: The evidence items do not provide a complete inventory of federal protections enacted through May 22, 2025.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Their stories about this are, in my view, greatly flawed, but having outside groups with science fiction chops writing stuff like this distracts away from those flaws, and gives more power to the very companies trying hardest to race towards AGI.”
Our reading: OpenAI's and Anthropic's AGI-imminence stories are greatly flawed, and outside scenarios like AI 2027 distract from those flaws and empower those companies.
No horizon Evidence as of
Limitation: The criticism of the company stories is underspecified and partly evaluative.
Assessment: UnresolvedSaid
Gary Marcus Observation
“And so far, anyway, the impact on the labor market of Generative AI has been modest; ditto for productivity in any measurable statistics.”
Our reading: So far, generative AI's impact on the labor market and on measurable productivity has been modest.
No horizon Evidence as of
Limitation: Geographic and occupational coverage is limited.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“AI 2027 isn’t slowing them down; it’s putting wind (and money, and political power) in their sails.”
Our reading: AI 2027 is giving AI companies like OpenAI and Anthropic more money and political power rather than slowing them.
No horizon Evidence as of
Limitation: Investor motives and private political discussions may be unavailable.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“It’s also encouraging the world to make short-term choices about AI (e.g., making plans around export controls when China will inevitably catch up) rather than longer-term choices (investing in research to develop safer, more alignable approaches to AI).”
Our reading: China will inevitably catch up in AI despite export controls.
No deadline stated Evidence as of
Limitation: The interpretation of catch up materially affects the result.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Tall tales about the imminence of AGI aren’t slowing down the AI race dynamic that the authors of AI-2027 want to mitigate; they are speeding that very dynamic up.”
Our reading: Stories claiming AGI is imminent are speeding up, not slowing, the AI race dynamic.
No horizon Evidence as of
Limitation: The accessible experimental evidence measures attention, not development decisions.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Cell phones and the internet were pretty big, but it’s not obvious their impact was greater than the industrial revolution, and so far AI’s impact has (despite all the PR) been far less than either cell phones or the internet.”
Our reading: So far, AI's impact has been far less than that of cell phones or the internet.
No horizon Evidence as of
Limitation: The evidence provides no common measure of accumulated impact or threshold for 'far less'.
Assessment: UnresolvedSaid
Gary Marcus Observation
“We have no concretely implemented international AI governance, no national AI agency; we are no longer even positioned well to detect and address AI-escalated cybercrime.”
Our reading: As of May 2025, there is no concretely implemented international AI governance and no US national AI agency, and the US is no longer even well positioned to detect and address AI-escalated cybercrime.
No horizon Evidence as of
Limitation: The OECD article is retrospective and carries an authors' views disclaimer.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: UnresolvedSaid
Gary Marcus Observation
“By many accounts, AI-fueled scams have exploded, too, and again there is no coherent federal response.”
Our reading: By many accounts, AI-fueled scams have exploded, and there is no coherent US federal response to them.
No horizon Evidence as of
Limitation: Vendor detection data can change with customer coverage and detection methods.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Because both countries are both building essentially the same kind of models with the same kinds of techniques using the same kinds of data, the results from the two nations are essentially converging on the same outcomes.”
Our reading: US and Chinese AI results are converging because both use the same kinds of models, techniques, and data.
No horizon Evidence as of
Limitation: A supported assessment of benchmark convergence alone would drop the claim's causal commitment.
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“With no regulatory pressure, tech companies would have little incentive to prioritize safety, transparency, or ethical design; any costs to society would be borne by society.”
Our reading: With no regulatory pressure, tech companies would have little incentive to prioritize safety, transparency, or ethical design, and any costs to society would be borne by society.
No horizon Evidence as of
Only if: with no regulatory pressure
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“First, if current trends continue, the “race” with China will not end in victory, for either side.”
Our reading: If current trends continue, neither the US nor China will win the AI race.
No deadline stated Evidence as of
Only if: if current trends continue
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Gary Marcus Observation
“So-called leaderboards are no longer dominated by any one country.”
Our reading: AI leaderboards are no longer dominated by any single country.
No horizon Evidence as of
Limitation: Several categories from one benchmark operator are not independent measures.
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Gary Marcus Forecast
“With enough money you can have TAs and smaller sections of say 20 students each, taught by teaching assistants, but with university budgets being slashed, few universities will have enough money to make that work.”
Our reading: With university budgets being cut, few universities will have enough money to replace term papers with small TA-led sections.
No deadline stated Evidence as of
Limitation: US universities are an interpretive population choice.
Source: ”Everyone is cheating their way through college” with GenAI. Who should bear the costs?
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“GenAI has nuked all of this.”
Our reading: GenAI has destroyed term papers' role in giving writing practice, prompting reflection and evaluating students at scale.
No horizon Evidence as of
Limitation: The strongest negative experiment examines take-home examinations rather than term papers.
Source: ”Everyone is cheating their way through college” with GenAI. Who should bear the costs?
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“The creators and distributors of GenAI, who have brought this huge problem to society?”
Our reading: The creators and distributors of GenAI have brought this huge problem, GenAI undermining college education and evaluation, to society.
No horizon Evidence as of
Limitation: The strongest direct experiment covers five psychology modules at one UK university.
Source: ”Everyone is cheating their way through college” with GenAI. Who should bear the costs?
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“(Those systems also can’t reliably stick to the rules of chess despite having them – and millions of games – in their database, a manifestation of the related problem of extracting statistical tendencies without every fully deriving and apprehending the correct abstractions).”
Our reading: LLMs can't reliably stick to the rules of chess despite having the rules and millions of games in their data, and this is a manifestation of the related problem of extracting statistical tendencies without ever fully deriving and grasping the correct abstractions.
No horizon Evidence as of
Limitation: Puzzle results, full-game results, and playing strength are not interchangeable.
Assessment: UnresolvedSaid
Gary Marcus Observation
“They don’t have a database of records like any proper business would (which would be a strong basis to solve the problem); and they don’t have what people like Yann LeCun or Judea Pearl or I would call a world model.”
Our reading: LLMs don't have a database of records like any proper business would, and they don't have what people like Yann LeCun, Judea Pearl, or Marcus would call a world model.
No horizon Evidence as of
Limitation: One documented architecture cannot establish a claim about all LLMs.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“We will eventually get to something better, but continuing to put all our eggs in the Henrietta’s LLM basket is absurd to the point of being delusional.”
Our reading: AI will eventually get to something better than LLMs.
No deadline stated Evidence as of
Limitation: The result depends on how broadly 'something better' is read.
Assessment: UnresolvedSaid
Gary Marcus Observation
“And it would be absurd to say that o3 can do all tasks that humans can do in 1.7 hours or that GPT-3 could do anything that a person can do in 15 seconds”
Our reading: o3 cannot do all tasks humans can do in 1.7 hours, and GPT-3 could not do everything a person can do in 15 seconds.
No horizon Evidence as of
Limitation: Unresolved reflects an evidence gap, not evidence that either model has universal capability.
Source: The latest AI scaling graph - and why it hardly makes sense
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Claude does what LaMDA does better because it has more data, but I don’t see any really argument that Claude is any more sentient than a web browser.”
Our reading: Claude does what LaMDA does better because it has more data, but Marcus says he does not see any real argument that Claude is any more sentient than a web browser.
No horizon Evidence as of
Limitation: Private training information limits causal assessment.
Source: New adventures in AI hype: “Our language models are so ‘conscious’ we need to give them rights”
Assessment: UnresolvedSaid
Gary Marcus Observation
“What these systems do, no more and no less, is to put together sequences of words, but without any coherent understanding of the world behind them, like foreign language Scrabble players who use English words as point-scoring tools, without any clue about what that mean.”
Our reading: LLMs only string together sequences of words and have no coherent understanding of the world behind them.
No horizon Evidence as of
Limitation: The positive mechanistic evidence comes from the model developer and has acknowledged methodological limits.
Source: New adventures in AI hype: “Our language models are so ‘conscious’ we need to give them rights”
Assessment: UnresolvedSaid
Gary Marcus Observation
“NVidia is down significantly more this year (23.4%) than the S&P (11.5%).”
Our reading: NVidia's stock is down 23.4% in 2025 so far, significantly more than the S&P's 11.5% decline.
No horizon Evidence as of
Limitation: Primary historical records sought were inaccessible or returned the wrong period.
Source: Poor ROI for GenAI
Assessment: UnresolvedSaid
Gary Marcus Observation
“Real world customers, many of whom are disappointed, are rarely mentioned.”
Our reading: Real-world customers, many of whom are disappointed, are rarely mentioned in the pundits' discussion of AI progress.
No horizon Evidence as of
Limitation: The survey was commissioned by an AI vendor and measures adoption disappointment, which can include organizational problems as well as product shortcomings.
Source: Deep Learning, Deep Scandal
Assessment: UnresolvedSaid
Gary Marcus Observation
“Deep learning is indeed finally hitting a wall, in the sense of reaching a point of diminishing results.”
Our reading: Deep learning is hitting a wall in the sense of reaching diminishing returns.
No horizon Evidence as of
Limitation: METR evaluated an earlier GPT-4.5 checkpoint in a short, preliminary evaluation.
Source: Deep Learning, Deep Scandal
Assessment: UnresolvedSaid
Gary Marcus Observation
“An LLM just spits back (sometimes in transmogrified form) what it has read.”
Our reading: An LLM just spits back what it has read, sometimes in transmogrified form.
No horizon Evidence as of
Limitation: Unresolved does not mean that the evidence favors the claim.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Humans are (usually, if well-motivated) smart enough to sort things like jokes and fiction from reality, but current AI doesn’t have a clue.”
Our reading: Current AI systems cannot tell jokes and fiction apart from reality, while well-motivated humans usually can.
No horizon Evidence as of
Limitation: The cited studies do not establish either representative failures or a usable ability on the specified fiction-versus-fact task during the assessment period.
Assessment: UnresolvedSaid
Gary Marcus Observation
“I do genuinely believe today’s Turing Test takers are better AI systems than an earlier generation. I hardly think that means anything is “over”.”
Our reading: Today's Turing Test-taking AI systems are better than the earlier generation, but that does not mean anything is 'over'.
No horizon Evidence as of
Limitation: This is an evidence gap, not evidence that contemporary systems lack broader improvements.
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“But I am willing to wager that if someone put in that kind of effort and carefully built out the test, no current system would come close to human performance.”
Our reading: If a careful comprehension challenge test were built, no current AI system would come close to human performance on it.
No deadline stated Evidence as of
Only if: if someone carefully built out the comprehension challenge test
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: UnresolvedSaid
Gary Marcus Observation
“Unfortunately, nobody has ever fully implemented the comprehension challenge test that I proposed.”
Our reading: Nobody has fully implemented Marcus's proposed comprehension challenge test.
No horizon Evidence as of
Limitation: The boundary between a related benchmark and full implementation is not formally defined.
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Agents will be a disappointment, precisely because of all of the ongoing reliability and hallucination issues.”
Our reading: AI agents will be a disappointment because of ongoing reliability and hallucination problems.
No deadline stated Evidence as of
Limitation: The evidence mixes enterprise decision-makers, agent developers and deployed-system practitioners.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“For most GenAI model building companies, profits will be meager.”
Our reading: Most GenAI model-building companies will have meager or no profits.
No deadline stated Evidence as of
Limitation: The evidence concerns one company's projected losses before the claim was published, not comparable realized results for a defined population [e1].