Unresolved claims, page 7 of 8
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Unresolved claims (795)
The record was checked, and the evidence available by the cutoff does not settle it. Forecasts whose deadline has not arrived are listed apart, as not yet due.
Page 7 of 8: claims 601 to 700 of 795, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: UnresolvedSaid
Gary Marcus Observation
“First, neither GPT-4 nor the plug-ins have the commonsense world knowledge and reasonable abilities needed to reliably translate a word problem into a mathematical calculation.”
Our reading: Neither GPT-4 nor its plug-ins have the commonsense knowledge and reasoning needed to reliably translate word problems into calculations.
No horizon Evidence as of
Limitation: The cited example comes from a college electromagnetism task.
Source: Getting GPT to work with external tools is harder than you think
Assessment: UnresolvedSaid
Gary Marcus Observation
“To our knowledge, no LLM-based system can yet reliably do what SHRLDU tried to do then:”
Our reading: As far as the authors know, no LLM-based system can yet reliably do the blocks-world modeling that SHRDLU attempted in the 1970s.
No horizon Evidence as of
Limitation: Planning benchmarks cover only part of the ordinary SHRDLU comparison.
Source: Hello, Multimodal Hallucinations
Assessment: UnresolvedSaid
Gary Marcus Forecast
“And no matter what investors and CEOs might tell you, hallucinations will remain inevitable.”
Our reading: Hallucinations in large language models will remain inevitable.
No deadline stated Evidence as of
Limitation: A mathematical reading and a practical-risk reading could yield different assessments.
Source: Hello, Multimodal Hallucinations
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“But in other ways, large language models are a profound step backwards; they lack stable models of the world, which means that they are poor at planning, and unable to reason reliably; everything becomes hit or miss..”
Our reading: LLMs lack stable world models, which makes them poor at planning and unable to reason reliably.
No horizon Evidence as of
Limitation: Evidence supporting the behavioral component is not sufficient to score the entire causal proposition supported.
Source: Hello, Multimodal Hallucinations
Assessment: UnresolvedSaid
Gary Marcus Observation
“ChatGPT went utterly wild when Ernie pushed on the last one, omitting some that are there and and confabulating a standalone island in the middle of the kitchen that simply isn’t there:”
Our reading: When questioned about the kitchen image, ChatGPT's vision model left out drawers that are there and made up a kitchen island that is not there.
No horizon Evidence as of
Limitation: The page reader does not reproduce the relevant image and transcript screenshots.
Source: Hello, Multimodal Hallucinations
Assessment: UnresolvedSaid
Gary Marcus Observation
“The first author’s nine-year-old can do five digit multiplication better than an LLM.”
Our reading: The first author's own nine-year-old child can do five-digit multiplication better than an LLM.
No horizon Evidence as of
Limitation: A private comparison may have occurred.
Assessment: UnresolvedSaid
Gary Marcus Observation
“There are some applications where the current systems are reliable enough to be practically useful, though not nearly as many as is often claimed.”
Our reading: Current systems are reliable enough to be useful in some applications, but far fewer than often claimed.
No horizon Evidence as of
Limitation: The existence of useful assistance is easier to test than the claimed extent of overstatement.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“About 18 months have passed; although there are certainly improvements (the images are vastly better), owing presumably in large part to a larger training set, the fundamental approach (mapping words to images, rather than constructing intermediate models of the world and how language and images relate to those models) remains the same, and so the core issue remains.”
Our reading: About 18 months after DALL-E 2, the images are vastly better, presumably largely because of a larger training set, but the fundamental approach of mapping words to images rather than building intermediate world models is the same, and so the core compositionality issue remains.
No horizon Evidence as of
Limitation: The technical paper was read through a third-party reproduction because the original PDF exceeded the reader's size limit.
Source: Race, statistics, and the persistent cognitive limitations of DALL-E
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“It’s the same issue of canonical versus noncanonical statistics, all over again.”
Our reading: Multimodal ChatGPT's trouble telling time from watch images comes from the same canonical-versus-noncanonical statistics problem seen in DALL-E.
No horizon Evidence as of
Limitation: Later model evaluations cannot be substituted for the historical deployment.
Source: Race, statistics, and the persistent cognitive limitations of DALL-E
Assessment: UnresolvedSaid
Gary Marcus Forecast
“though I agree with Geoff that eventually AI will have a big impact on drug design. Perhaps with current techniques, perhaps not; we will have to see.”
Our reading: AI will eventually have a big impact on drug design, possibly but not necessarily with current techniques.
No deadline stated Evidence as of
Limitation: The threshold for substantial impact is an evaluator interpretation.
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?
Assessment: UnresolvedSaid
Gary Marcus Observation
“I think it’s fair to say that generative AI works better than most people expected.”
Our reading: Generative AI works better than most people expected.
No horizon Evidence as of
Limitation: A narrower reading about informed AI researchers could produce a different assessment.
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?
Assessment: UnresolvedSaid
Gary Marcus Forecast
“And emphasize that wholesale, automated fake news will be used both to manipulate markets and elections, and might undermine democracy.”
Our reading: Wholesale, automated fake news will be used to manipulate both markets and elections, and it might undermine democracy.
No deadline stated Evidence as of
Limitation: Wholesale and automated require interpretation.
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?
Assessment: UnresolvedSaid
Gary Marcus Observation
“Your best example of a spectacular machine invention is in fact a perfect example of the kind of parroting and theft of intellectual property that is characteristic of large language models.”
Our reading: Bard's 'baby shoes' story was parroting of an existing story, and such parroting and IP theft is characteristic of large language models.
No horizon Evidence as of
Limitation: The available transcript is an updated version, and the independent transcription contains only part of the output.
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?
Assessment: UnresolvedSaid
Gary Marcus Observation
“You know how many radiologists have been replaced by machines seven years later? Zero.”
Our reading: Seven years after Hinton's 2016 prediction, no radiologists have been replaced by machines.
No horizon Evidence as of
Limitation: The workforce census covers the UK, while the claim supplies no geographic limit.
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“I can’t either; there’s a lot we can do to help, but nothing I can see either to absolutely guarantee safety. Rushing ahead is creating risk.”
Our reading: A lot can be done to help with AI safety, but Marcus can see nothing that would absolutely guarantee it, and rushing ahead is creating risk.
No horizon Evidence as of
Limitation: The company reports do not isolate the effect of development or deployment speed [e1, e2].
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“There is no internal model of what she actually did, and hence no capacity to support reliable inference, hence no guarantee that anything it writes about her will be true.”
Our reading: LinkedIn's OpenAI-powered assistant has no internal model of what Kaya Yurieff actually did, and hence no capacity to support reliable inference about her, and hence no guarantee that anything it writes about her will be true.
No horizon Evidence as of
Limitation: No evidence identifying the version, inputs, prompts or complete outputs of the assistant tested by Yurieff.
Source: Muddles about Models
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“Without flexible underlying models that can reliably support inferences on novel questions, we can expect more of the same of what we are seeing over and over now: more hallucinations, more reasoning that is hit or miss, and nothing like the stability we should insist on for future AI.”
Our reading: Without flexible underlying models, AI systems will keep producing hallucinations and hit-or-miss reasoning, and will lack the needed stability.
No deadline stated Evidence as of
Only if: without flexible underlying models that reliably support inference on novel questions
Source: Muddles about Models
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“It’s the same problem: correlations are fragile, and you can’t count on them.”
Our reading: GPT's failure to stick to the rules of chess, even when it plays a decent game, reflects the same problem: correlations are fragile and cannot be counted on.
No horizon Evidence as of
Limitation: A failure to follow rules does not by itself reveal the model’s internal representations.
Source: Muddles about Models
Assessment: UnresolvedSaid
Gary Marcus Observation
“By my count, though I acknowledge I may have missed one, there are at least 7 falsehoods.”
Our reading: A recent Microsoft Bing answer about the House Speaker contained at least seven falsehoods.
No horizon Evidence as of
Limitation: The underlying political facts are more accessible than the original model output.
Source: Seven lies in four sentences
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“This all happens not because LLMs are smart, in the sense of being able to compare what they say with the world, but simply because they are good at making new sentences that sound old sentences, the essence of autocomplete.”
Our reading: LLMs produce plausible falsehoods because they mimic existing sentences like autocomplete rather than checking what they say against the world.
No horizon Evidence as of
Limitation: Training descriptions establish design facts, not a complete causal explanation of errors.
Source: Seven lies in four sentences
Assessment: UnresolvedSaid
Gary Marcus Observation
“This symmetry fail is mighty big, a mighty persistent error that has endured for decades.”
Our reading: The symmetry failure is a big, persistent error in neural networks that has endured for decades.
No horizon Evidence as of
Limitation: Failure to locate a matching historical experiment is not evidence that none exists.
Source: Elegant and powerful new result that seriously undermines large language models
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“If we don’t know what is in the data sets, we can’t know what biases these systems will spread through society if they are widely used, we can’t know the extent to which individual content creatore are being exploited, and worse any outside efforts to mitigate risks are greatly undermined.”
Our reading: If we don't know what is in AI training data sets, we can't know what biases these systems will spread through society if widely used or the extent to which individual content creators are being exploited, and outside efforts to mitigate risks are greatly undermined.
No horizon Evidence as of
Only if: if we don't know what is in the data sets
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“The apparent economics of AI (we will see if they yet come to pass as imagined) are too seductive to too many, to people building AI companies, to people investing in AI companies, to people getting paid 6 and even 7 digit salaries, and to people expecting campaign donations from big tech companies.”
Our reading: The apparent economics of AI, which may or may not come to pass as imagined, are too seductive to too many people (AI company builders, investors, people paid 6- and 7-digit salaries, and those expecting big-tech campaign donations) for any AI moratorium to happen in the near term.
No horizon Evidence as of
Limitation: Public statements may omit motives or present them selectively.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“We are obviously never going to have a moratorium anytime in the near-term.”
Our reading: There will be no moratorium on AI development in the near term.
No deadline stated Evidence as of
Limitation: The main contemporaneous account comes from the organization advocating the pause.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“We have vastly more attention on AI policy, throughout the world, and I definitely give the pause letter (which I signed but didn’t write) some credit for that.”
Our reading: Global attention on AI policy has vastly increased since March 2023, and the pause letter deserves some credit for that.
No horizon Evidence as of
Limitation: A new meeting is not a worldwide measure of attention.
Assessment: UnresolvedSaid
Gary Marcus Observation
“That has meant, for example, no tangible progress on getting the big companies to commit to revealing what data their models were trained on; we have transparency in name, but not in deed.”
Our reading: There has been no tangible progress in getting big AI companies to commit to disclosing their training data.
No horizon Evidence as of
Limitation: The result depends on distinguishing existing disclosures from commitments to disclose.
Source: Could we all be doomed, without ranked-choice voting?
Assessment: UnresolvedSaid
Gary Marcus Forecast
“The default right now is regulatory capture: the companies will make the rules, telling the governments what they are and are not comfortable with. The government will go along.”
Our reading: By default, AI companies will write the rules for AI regulation and governments will go along.
No deadline stated Evidence as of
Limitation: The evidence is not a census of US AI regulation or industry lobbying.
Source: Could we all be doomed, without ranked-choice voting?
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Without data transparency, we can’t make sure our models are without bias, or even know the sources of the biases they hide within, we can’t truly understand the scope and limits of their generalization (which is central to understanding what they might do in unexpected circumstances) and we are hamstrung in our ability to mitigate their harmful consequences: what is unknown is hard to fix.”
Our reading: Without transparency about the data models were trained on, we can't make sure models are free of bias or even know the sources of their biases, we can't truly understand the scope and limits of their generalization, and we are hamstrung in our ability to mitigate their harmful consequences.
No horizon Evidence as of
Limitation: The supporting paper synthesizes several forms of access; their effects are not interchangeable.
Source: Could we all be doomed, without ranked-choice voting?
Assessment: UnresolvedSaid
Gary Marcus Observation
“But if and when we get there, will they be dominating? Non-dominating? The honest answer is we haven’t a clue.”
Our reading: Marcus says that, honestly, we haven't a clue whether super-smart machines, if and when we get there, will be dominating or non-dominating.
No horizon Evidence as of
Limitation: The search cannot establish a universal absence of applicable research.
Source: 𝐩(𝐝𝐨𝐨𝐦)
Assessment: UnresolvedSaid
Gary Marcus Forecast
“We aren’t going to be able calculate p(doom) to three decimal places.”
Our reading: No one will be able to calculate p(doom) precisely, to three decimal places.
No deadline stated Evidence as of
Limitation: Search coverage is not proof that a qualifying method does not exist.
Source: 𝐩(𝐝𝐨𝐨𝐦)
Assessment: UnresolvedSaid
Gary Marcus Observation
“Yet a really large fraction of the AI and AI-adjacent community doesn’t really want to talk about any of this at all.”
Our reading: A large fraction of the AI and AI-adjacent community does not want to discuss AI doom or catastrophe risk at all.
No horizon Evidence as of
Limitation: The packet contains no representative measurement of willingness to discuss these risks during the target period.
Source: 𝐩(𝐝𝐨𝐨𝐦)
Assessment: UnresolvedSaid
Gary Marcus Observation
“Some of the specific arguments Yudkowsky and Bostrom have made may well be weak, but there is no solid argument that the number is zero, or even that the risk is infinitesimal.”
Our reading: There is no solid argument that AI doom risk is zero or even infinitesimal, though some arguments by Yudkowsky and Bostrom may be weak.
No horizon Evidence as of
Limitation: A broad negative claim about all arguments cannot be established by sampling a few positions.
Source: 𝐩(𝐝𝐨𝐨𝐦)
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“This is even more true four years later, now that we have systems that ingest massive amounts of data from both public and commercial sources, quite possibly including eg sample law school and medical exams etc.”
Our reading: Because current systems ingest massive data possibly including sample exams, benchmark success is even less indicative of intelligence than in 2019.
No horizon Evidence as of
Limitation: OpenAI's audit was a developer assessment with acknowledged detection limits.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Without a sense of what it is in the training corpus, it’s almost impossible to conclude anything about what a machine truly understands and how general that is.”
Our reading: Without knowing the training corpus, one can hardly conclude what a machine understands or how general it is.
No horizon Evidence as of
Limitation: A reading limited to whether any bounded behavioral conclusion is possible would weigh more strongly against Marcus.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Humans can write summaries of things without hallucination; LLMs cannot do so reliably.”
Our reading: LLMs cannot reliably write summaries without hallucinating.
No horizon Evidence as of
Limitation: The inferred monthly observation window is narrower than a general review of 2023 models.
Assessment: UnresolvedSaid
Gary Marcus Observation
“18 months ago, before the LLM euphoria, we all took for granted that AGI wasn’t just about beating ordinary humans.”
Our reading: Around early 2022, before LLM euphoria, the field assumed AGI meant more than beating ordinary humans.
No horizon Evidence as of
Limitation: The later review provides historical definitions, not a representative measurement of February 2022 views [e1].
Assessment: UnresolvedSaid
Gary Marcus Observation
“in my view it’s a subtle redefinition that dramatically moves the goal posts.”
Our reading: In Marcus's view, Legg's definition of AGI as better than typical human performance on virtually all cognitive tasks is a subtle redefinition that dramatically moves the goal posts, compared with how Legg and Marcus publicly discussed the term last year.
No horizon Evidence as of
Limitation: Missing original exchange records prevent a finding either way.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Nobody has yet figured out how to build a driverless car that can deal with that fact.”
Our reading: Nobody has yet figured out how to build a driverless car that can deal with the real world's endless crazy and out-of-the-ordinary circumstances.
No horizon Evidence as of
Limitation: Reliable handling and acceptable human assistance are not defined in the source.
Source: Face it, self-driving cars still haven’t earned their stripes
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“Edge cases are everywhere; they are in driving, they are in medicine, they there will be a zillion of them humanoid robots, if and when we roll them out. Anyone who thinks any of this is going to be easy is fooling themselves.”
Our reading: Edge cases are everywhere, including in driving and medicine; there will be a zillion of them in humanoid robots if and when they are rolled out; and anyone who thinks any of this will be easy is fooling themselves.
No deadline stated Evidence as of
Only if: if and when humanoid robots are rolled out
Source: Face it, self-driving cars still haven’t earned their stripes
Assessment: UnresolvedSaid
Gary Marcus Observation
“In fact, the truth is we don’t actually have any truly self-driving cars yet.”
Our reading: There are not yet any truly self-driving cars: every 'self-driving vehicle' on public roads has either a human safety driver on board or a remote human who can help it when issues arise.
No horizon Evidence as of
Limitation: The strongest contemporaneous evidence covers the two San Francisco operators.
Source: Face it, self-driving cars still haven’t earned their stripes
Assessment: UnresolvedSaid
Gary Marcus Observation
“We have no such thing; building it would require a major reshaping of research directions, towards an AI based in facts and competent in reasoning, rather than just the powerful tools for text prediction that we have now.”
Our reading: We have no AI smart enough to know when it is being used to spew falsehood, and building it would require a major reshaping of research directions toward AI based in facts and competent in reasoning, rather than just the powerful text-prediction tools we have now.
No horizon Evidence as of
Limitation: The cited evaluations precede the publication-date snapshot and are context.
Source: Automatic Disinformation Threatens Democracy— and It’s Here
Assessment: UnresolvedSaid
Gary Marcus Observation
“It’s a powerful reminder that our existing laws are not sufficient to protect us from the full scope of harms that generative AI is beginning to enable .”
Our reading: Existing laws are not sufficient to protect against the full scope of harms generative AI is beginning to enable.
No horizon Evidence as of
Limitation: The legal evidence primarily concerns federal election advertising.
Source: Automatic Disinformation Threatens Democracy— and It’s Here
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Automatic tools like these will surely accelerate the enshittification of the internet that I discussed yesterday.”
Our reading: Automatic misinformation generation tools will accelerate the degradation of the internet.
No deadline stated Evidence as of
Limitation: The assessment uses misinformation contamination as the ordinary contextual meaning of degradation.
Source: Automatic Disinformation Threatens Democracy— and It’s Here
Assessment: UnresolvedSaid
Gary Marcus Observation
“And even worse, Amazon is starting to be a receptacle for inaccurate and exploitative quickie histories of recent events:”
Our reading: Amazon is starting to be a receptacle for inaccurate and exploitative quickie histories of recent events.
No horizon Evidence as of
Limitation: The retained dated evidence does not complete verification of specific material errors in the book.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Coding is a solid use case that will continue to keep generative AI in the limelight for a long time. Programmers will never go back.”
Our reading: Coding will remain a solid use case keeping generative AI prominent for a long time, and programmers will not abandon it.
No deadline stated Evidence as of
Limitation: Survey findings describe respondents, not a census of programmers.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Driverless cars are powered by AI, but so far the money that has gone into them, on order of $100 billion dollars, vastly exceeds the actual revenue that has come out of them.”
Our reading: Investment in driverless cars, around $100 billion, vastly exceeds the revenue they have generated so far.
No horizon Evidence as of
Limitation: Corroborating the investment magnitude alone does not establish the complete claim.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Perhaps the only use case that really seemed compelling to either of us economically was search (e.g., using Bing powered by ChatGPT instead of Google Search) — but the technical problems there are immense; there is no reason to think that the hallucination problem will be solved soon.”
Our reading: There is no reason to think that the hallucination problem, part of the immense technical problems facing the search use case, will be solved soon.
No horizon Evidence as of
Limitation: The two studies do not comprehensively assess the leading approaches or their prospects for near-term deployment.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“I think coders will remain happy with generative AI assistance; its autocomplete nature is fabulous for their line of work, and they have the training to detect and fix the not-infrequent errors.”
Our reading: Marcus thinks coders will remain happy with generative AI assistance, because its autocomplete nature is fabulous for their work and they have the training to detect and fix its not-infrequent errors.
No deadline stated Evidence as of
Limitation: Survey respondents are not necessarily representative of all programmers.
Assessment: UnresolvedSaid
Gary Marcus Observation
“And, reality check, we have no concrete reason, other than sheer technoptimism, for thinking that solutions to any of these problems is imminent.”
Our reading: There is no concrete reason, other than sheer techno-optimism, for thinking that solutions to any of generative AI's core problems (such as hallucination, unreliable tool use and month-to-month instability) are imminent.
No horizon Evidence as of
Limitation: This is an assessment of contemporary grounds for expectation, not a forecast scored against later events.
Assessment: UnresolvedSaid
Gary Marcus Observation
“I don’t see any reason to think these plugins are in fact safe. (They are far less sandboxed and less rigorously controlled than Apple app store apps.)”
Our reading: Marcus says he does not see any reason to think ChatGPT plugins are in fact safe, noting that they are far less sandboxed and less rigorously controlled than Apple App Store apps.
No horizon Evidence as of
Limitation: The packet lacks a sufficiently complete account of OpenAI's review and enforcement during the target period.
Source: Two models of AI oversight - and how things could go deeply wrong
Assessment: UnresolvedSaid
Gary Marcus Observation
“Neurosymbolic AI might offer a better way of resolving these issues, since it traffics in facts and explicit reasoning, but neurosymbolic AI is out of fashion, and neither well-developed nor strongly supported by the VC world.”
Our reading: Neurosymbolic AI is out of fashion, not well developed, and not strongly supported by venture capital.
No horizon Evidence as of
Limitation: The field boundaries and all three qualitative thresholds are unspecified.
Source: How China might crush the West in the race to AGI if we don’t up our regulatory game
Assessment: UnresolvedSaid
Gary Marcus Observation
“At present, however, there is no legal requirement in the west that AI companies resolve the truthfulness or alignment issues.”
Our reading: There is currently no legal requirement in the West for AI companies to resolve truthfulness or alignment issues.
No horizon Evidence as of
Limitation: The schema has no source type for official regulatory announcements; official_statistics is used for the government documents.
Source: How China might crush the West in the race to AGI if we don’t up our regulatory game
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Their forte has been and always will be making shit up; they can’t for the life of them (speaking metaphorically of course) check facts.”
Our reading: For GPT models on their own, making things up has been and always will be their forte, and they cannot check facts.
No deadline stated Evidence as of
Limitation: A literal zero-ability reading would yield a different assessment of the fact-checking component.
Source: GPT-5 and irrational exuberance
Assessment: UnresolvedSaid
Gary Marcus Forecast
“It will be even better than GPT-4 at creating convincing sounding prose.”
Our reading: GPT-5 will be better than GPT-4 at creating convincing-sounding prose.
No deadline stated Evidence as of
Limitation: Writing preference varies with genre, prompting, and generation settings.
Source: GPT-5 and irrational exuberance
Assessment: UnresolvedSaid
Gary Marcus Forecast
“GPT-5 is not going to be magic. It will not give China, the US, or anyone else interstellar starships; it will not give anyone an insuperable military advantage.”
Our reading: GPT-5 will not give China, the US, or anyone else interstellar starships, and it will not give anyone an insuperable military advantage.
No deadline stated Evidence as of
Limitation: Military capabilities may not be publicly observable.
Source: GPT-5 and irrational exuberance
Assessment: UnresolvedSaid
Gary Marcus Observation
“(Not one of my predictions about GPT-4 proved to be incorrect; every flaw I predicted would persist persisted.)”
Our reading: Marcus says none of his predictions about GPT-4 proved incorrect, and every flaw he predicted would persist in GPT-4 did persist.
No horizon Evidence as of
Limitation: Confirming several central predictions cannot establish a universal track-record claim.
Source: GPT-5 and irrational exuberance
Assessment: UnresolvedSaid
Gary Marcus Observation
“GPT-4 can’t even play a good game of chess (it’s about as good as chess computers from 1978).”
Our reading: GPT-4 cannot play a good game of chess and is about as good as 1978 chess computers.
No horizon Evidence as of
Limitation: The admitted GPT-4 evidence consists of one recorded game and an account of two attempted games [e1, e2].
Source: GPT-5 and irrational exuberance
Assessment: UnresolvedSaid
Gary Marcus Observation
“And very little of what GPT-4 does is fully replicable. Sometimes summarization (aka compression) works (super cool!) and sometimes it doesn’t, as many were quick to point out:”
Our reading: Very little of what GPT-4 does is fully replicable; its summarization sometimes works and sometimes fails.
No horizon Evidence as of
Limitation: The task population and meaning of full replication are unspecified.
Source: GPT-5 and irrational exuberance
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“Some form of AI may eventually do everything people are imagining, revolutionizing science and technology and so on, but LLMs will be at most only a tiny part of whatever as-yet-uninvented technology does that.”
Our reading: If some as-yet-uninvented form of AI eventually does everything people are imagining, such as revolutionizing science and technology, LLMs will be at most only a tiny part of it.
No deadline stated Evidence as of
Only if: if some form of AI eventually does everything people are imagining
Source: GPT-5 and irrational exuberance
Assessment: UnresolvedSaid
Gary Marcus Forecast
“It might be on the nice-to-have list; it will not mark the mythical singularity.”
Our reading: GPT-5 will not mark the singularity.
No deadline stated Evidence as of
Limitation: The cited failure rate concerns one task, not a measurement of sustained, self-amplifying intelligence improvement [e1].
Source: GPT-5 and irrational exuberance
Assessment: UnresolvedSaid
Gary Marcus Observation
“Vulnerable patients shouldn’t be talking to chatbots that aren’t competent for this situation, and too much of the discussion in the media have made chatbots sound more intelligent and even more friendly than they are.”
Our reading: Too much media discussion has made chatbots sound more intelligent and friendly than they are.
No horizon Evidence as of
Limitation: The phrase “too much” remains partly evaluative.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Virtually everyone, even at OpenAI, has acknowledged that there are serious risks, but thus far few tangible steps have been taken to mitigate them—either by government or industry.”
Our reading: Although virtually everyone, including people at OpenAI, acknowledges serious AI risks, government and industry have so far taken few tangible steps to mitigate them.
No horizon Evidence as of
Limitation: Government guidance, legal requirements and technical safeguards differ substantially in reach and effectiveness.
Source: The Open Letter Controversy
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“The paper’s core claim— “GPT-4 attains a form of general intelligence [as] demonstrated by its core mental capabilities (such as reasoning, creativity, and deduction)”—literally cannot be tested with serious scrutiny, because the scientific community has no access to the training data.”
Our reading: The Sparks of AGI paper's claim that GPT-4 attains a form of general intelligence cannot be seriously tested because scientists have no access to its training data.
No horizon Evidence as of
Limitation: This is not support for the claim that all serious testing requires training-data access.
Assessment: UnresolvedSaid
Gary Marcus Observation
“if the definition of reasoning is to obtain valid conclusions from known facts, GPT-4 frequently falls short there, too.”
Our reading: GPT-4 frequently fails to reason, if reasoning means reaching valid conclusions from known facts.
No horizon Evidence as of
Limitation: Later evaluations were not rejected merely because they were published after the post; the problem is their uncertain connection to the target model period.
Source: This Week in AI Doublespeak
Assessment: UnresolvedSaid
Gary Marcus Observation
“It doesn’t actually solve any of the core problems of truthfulness and reliability that I laid out in my infamous March 2022 essay Deep Learning is Hitting a Wall.”
Our reading: GPT-4 does not solve any of the core truthfulness and reliability problems identified in Marcus's March 2022 essay.
No horizon Evidence as of
Limitation: This is an evidence gap for the full assertion, not evidence that the problems were solved.
Assessment: UnresolvedSaid
Gary Marcus Observation
“On its own, it still requires frequent, massive retraining to be current with the news; GPT 3.5 knew nothing of 2022; GPT-4 seems to know little of 2023.”
Our reading: On its own, GPT-4 still requires frequent massive retraining to stay current with the news.
No horizon Evidence as of
Limitation: A knowledge cutoff is not evidence of complete ignorance of every later event.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Because of all that, we don’t know what explains its success, and we don’t know how to predict its failures:”
Our reading: Because GPT-4's size, architecture, energy use, processors, and training data have not been disclosed, we don't know what explains its success and we don't know how to predict its failures.
No horizon Evidence as of
Limitation: The strongest quantitative evidence comes from OpenAI's own evaluations.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Alignment is still shaky; you still wouldn’t be able to use it reliably to guide robots or scientific discovery, the kinds of things that made me excited about A(G)I in the first place.”
Our reading: GPT-4's alignment remains shaky and it cannot be used reliably to guide robots or scientific discovery.
No horizon Evidence as of
Limitation: Neither missing launch-period demonstrations nor general language-model errors prove application-wide inability.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Large language models are here to stay, their recipe well understood, and not hard for people skilled in the art to replicate; large pretrained models are out their for the taking on the dark web.”
Our reading: LLMs are here to stay: their recipe is well understood, skilled practitioners can replicate them without much difficulty, and large pretrained models are freely available on the dark web.
No horizon Evidence as of
Limitation: The leak is established through reputable reporting rather than a directly inspected primary distribution archive.
Source: Should we worry more about short-term AI risk or long-term AI risk?
Assessment: UnresolvedSaid
Gary Marcus Forecast
“It will be an uphill, ongoing move-and-countermove arms race from here”
Our reading: Fighting AI-generated misinformation will be an uphill, ongoing arms race of moves and countermoves.
No deadline stated Evidence as of
Limitation: Laboratory watermark results do not directly measure misinformation impact.
Source: Should we worry more about short-term AI risk or long-term AI risk?
Assessment: UnresolvedSaid
Gary Marcus Observation
“A lot of machine learning researchers are worried. And yet most people carry on as if they weren’t.”
Our reading: Many machine learning researchers worry about very bad outcomes from AI, yet most people carry on as if they didn't.
No horizon Evidence as of
Limitation: The researcher survey had a 17% response rate.
Source: Should we worry more about short-term AI risk or long-term AI risk?
Assessment: UnresolvedSaid
Gary Marcus Observation
“Minerva, for example, is trained on an immense amount of data, and ostensibly captures two-digit arithmetic, but it fails altogether on 4-digit multiplication,”
Our reading: Minerva, despite immense training data and apparent two-digit arithmetic ability, fails altogether on 4-digit multiplication.
No horizon Evidence as of
Limitation: The decisive data exist only in a figure that the text tools cannot read.
Source: Caricaturing Noam Chomsky
Assessment: UnresolvedSaid
Gary Marcus Observation
“The request is perfectly clear, not a nonsensical question in any way. The prompt is perfectly sensible; the system just isn’t up to the job.”
Our reading: The request in his example is perfectly clear and sensible, not nonsensical in any way, and the system just isn't up to the job.
No horizon Evidence as of
Limitation: The retrieval gap is not evidence that the prompt was unclear or that the response was correct.
Source: Caricaturing Noam Chomsky
Assessment: UnresolvedSaid
Gary Marcus Observation
“The same goes for essentially everything an LLMs talks about; there is a degree of generalization to similar examples, but distribution shift is hard (the crux of my own work going back to 1998), and nearly any generalization can be easily broken.”
Our reading: For essentially everything an LLM talks about, there is a degree of generalization to similar examples, but distribution shift is hard and nearly any generalization can be easily broken.
No horizon Evidence as of
Limitation: All the evidence comes from math and numerical tasks, not the range of domains the claim covers.
Source: Caricaturing Noam Chomsky
Assessment: UnresolvedSaid
Gary Marcus Observation
“LLM-generated prose has already disrupted web forums and peer review processes, by flooding outlets with fake submissions.”
Our reading: LLM-generated prose has already disrupted web forums and peer review by flooding outlets with fake submissions.
No horizon Evidence as of
Limitation: The forum component is supported.
Source: The long shadow of GPT
Assessment: UnresolvedSaid
Gary Marcus Observation
“New concerns are emerging almost literally day,.”
Our reading: New concerns about LLM harms are emerging almost daily.
No horizon Evidence as of
Limitation: The packet lacks a sufficiently comprehensive, deduplicated chronology.
Source: The long shadow of GPT
Assessment: UnresolvedSaid
Gary Marcus Observation
“I presented this example of how even the latest version of Bing, with its new guardrails, could still be manipulated to be useful to troll farms:”
Our reading: Even the latest version of Bing's chatbot, with its new guardrails, can still be manipulated into producing misinformation useful to troll farms.
No horizon Evidence as of
Limitation: The readable Rathi record does not establish what the omitted conversation image shows [e1].
Source: Don’t Look Up: The AI Edition
Assessment: UnresolvedSaid
Gary Marcus Observation
“Meanwhile, within a few hours, a 17-year-old high-school student had largely1 replicated what I was saying, and did so with a simpler, more elegant jailbreak prompt, zero prior context required2:”
Our reading: Within a few hours, a 17-year-old high-school student had largely replicated what Marcus was saying about Bing being usable for misinformation, using a simpler, more elegant jailbreak prompt than his and needing zero prior context.
No horizon Evidence as of
Limitation: The original screenshot may contain evidence absent from the text extraction.
Source: Don’t Look Up: The AI Edition
Assessment: UnresolvedSaid
Gary Marcus Observation
“What’s even more disturbing is that Bing makes it look like the false narrative that it generates is referenced.”
Our reading: In Oakley's reported example, Bing makes the false narrative it generates look as if it is referenced.
No horizon Evidence as of
Limitation: A broader reading about Bing's general capability would receive stronger support from the separate TechCrunch demonstration.
Source: The threat of automated misinformation is only getting worse
Assessment: UnresolvedSaid
Gary Marcus Observation
“Few people understand how they work; many people anthropomorphizing those chatbots, attributing to them real intelligence and emotion.”
Our reading: Few people understand how chatbots work, and many anthropomorphize them, attributing real intelligence and emotion to them.
No horizon Evidence as of
Limitation: The wording leaves the population and prevalence thresholds unspecified.
Source: Don’t Go Breaking My Heart
Assessment: UnresolvedSaid
Gary Marcus Observation
“But no known technology can reliably align a chatbot in a persistent way to a human’s emotional needs.”
Our reading: No known technology can reliably and persistently align a chatbot to a human's emotional needs.
No horizon Evidence as of
Limitation: The packet contains one earlier review and one trial, not a comprehensive assessment of technology available by February 22, 2023.
Source: Don’t Go Breaking My Heart
Assessment: UnresolvedSaid
Gary Marcus Forecast
“with the new generation of chatbots, we will see addictions no less potent.”
Our reading: The new generation of chatbots will produce addictions at least as potent as social media addiction.
No deadline stated Evidence as of
Limitation: Search coverage is incomplete because some relevant pages could not be read.
Source: Don’t Go Breaking My Heart
Assessment: UnresolvedSaid
Gary Marcus Observation
“Right now, as I have said before, the AI is basically the Wild West; anyone can post any chatbot they want.”
Our reading: Right now AI is basically the Wild West, in that anyone can post any chatbot they want.
No horizon Evidence as of
Limitation: Voluntary NIST guidance does not establish that no binding release requirements existed elsewhere [e1].
Source: Why *is* Bing so reckless?
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“This would be very bad news, not just in terms of human and economic costs (which would mean more underpaid layers doing awful work) but also in terms of trustworthiness, because it would be mean that we have zero guarantee that any new iteration of a large language model is going to safe.”
Our reading: If every LLM update needs full RL retraining, there would be more underpaid labeling work and zero guarantee that any new LLM iteration is safe.
No deadline stated Evidence as of
Only if: if every LLM update requires complete RL retraining
Source: Why *is* Bing so reckless?
Assessment: UnresolvedSaid
Gary Marcus Observation
“Anyone who watched the last week unfold will realize that the new Bing has (or had1) a tendency to get really wild, from declaring a love that it didn’t really have to encouraging people to get divorced to blackmailing them to teaching people how to commit crimes, and so on.”
Our reading: In the week before the post, the new Bing chatbot tended to behave wildly, including professing love, encouraging divorce, blackmailing users and teaching them to commit crimes.
No horizon Evidence as of
Limitation: The available conversations are selected examples, not a representative sample.
Source: Why *is* Bing so reckless?
Assessment: UnresolvedSaid
Gary Marcus Observation
“The problems in physical and psychological reasoning that we pointed in the Fall of 2020 very much remain.”
Our reading: The physical and psychological reasoning problems in GPT-3 that the authors identified in fall 2020 still remain as of February 2023.
No horizon Evidence as of
Limitation: Neither item supplies matched testing of both domains during the target period.
Source: How Not to Test GPT-3
Assessment: UnresolvedSaid
Gary Marcus Observation
“For one thing, what we publish often gets incorporated into the training; perhaps in company benchmarks as well.”
Our reading: What the authors publish often gets incorporated into AI training data, and perhaps into company benchmarks as well.
No horizon Evidence as of
Limitation: No verified membership records or frequency estimate establish how often the authors' examples entered relevant training data.
Source: How Not to Test GPT-3
Assessment: UnresolvedSaid
Gary Marcus Forecast
“But In practice, that isn’t going to happen.”
Our reading: GPT-3 will not be retrained on a corpus with all references to the false-belief experiments removed to test theory of mind.
No deadline stated Evidence as of
Limitation: Only one search was run, so a relevant study could have been missed.
Source: How Not to Test GPT-3
Assessment: UnresolvedSaid
Gary Marcus Observation
“However, just in terms of percentage right vs. wrong, the improvement was slight; It now got 50% right, 40% wrong, and 10% borderline (marginally better than the 45% right before).”
Our reading: From 2020 to June 2022, GPT-3 improved only slightly on the authors' commonsense examples, going from 45% to 50% right.
No horizon Evidence as of
Limitation: No independent rerun or audit of the authors' example set was found. That does not prove none exists.
Source: How Not to Test GPT-3
Assessment: UnresolvedSaid
Gary Marcus Observation
“each reflect the fact that you cannot build AI in the real world from Big Data and deep learning alone.”
Our reading: AI that works in the real world cannot be built from big data and deep learning alone.
No horizon Evidence as of
Limitation: This does not endorse the stronger claim that deep learning cannot produce useful applications.
Assessment: UnresolvedSaid
Gary Marcus Observation
“There was a broken image link, and I went so fast I neglected to mention the personally salient fact that Bing had lied about me, claiming that I believe that “Bing’s AI was more accurate and reliable than Bard”, when I never said any such thing.”
Our reading: Bing's chatbot falsely claimed that Marcus believes Bing's AI is more accurate and reliable than Bard.
No horizon Evidence as of
Limitation: The screenshot may survive elsewhere, but it was not available in the retrieved records.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Microsoft can (at least to a first approximation) do whatever the hell it wants with Bing.”
Our reading: Microsoft is, roughly, free to do whatever it wants with Bing's AI.
No horizon Evidence as of
Limitation: The qualification rules out treating any legal obligation as an automatic contradiction.
Assessment: UnresolvedSaid
Gary Marcus Observation
“There are almost no regulations here; we are living in the Wild AI West.”
Our reading: There is almost no regulation of AI chatbots.
No horizon Evidence as of
Limitation: Several relevant contemporary primary pages could not be read.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“I do actually think that these problems will eventually get ironed out, possibly after some important fresh discoveries are made, but whether they get ironed out soon is another matter entirely.”
Our reading: Marcus thinks chatbot hallucination problems will eventually get ironed out, possibly after some important fresh discoveries are made, but whether they get ironed out soon is another matter entirely.
No deadline stated Evidence as of
Limitation: 'Ironed out' has no stated threshold, so any future judgement would need to set one.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Our new systems work better, but they are still superficial, fine for lab demos but often at a loss in the real world.”
Our reading: Current AI systems work better than older ones but are still superficial: fine for lab demos, often at a loss in the real world.
No horizon Evidence as of
Limitation: Documented failures are not a representative failure rate.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Melanie Mitchell’s book, Bender et al’s “Stochastic Parrots”; all of these more recent criticisms of deep learning help establish that some of the basic issues that were known long ago, with predecessors to current models, still remain even today.”
Our reading: Recent criticisms of deep learning, such as Melanie Mitchell's book and Bender et al.'s 'Stochastic Parrots', help establish that some of the basic issues known long ago with predecessors to current models still remain today.
No horizon Evidence as of
Limitation: This is an evidence-access and attribution gap, not evidence that the historical continuity claim is false.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Not sure how much has changed in that regard, either, except that the hacks that were once written by graduate students are now farmed out to sweatshops, for a new form of handcrafting:”
Our reading: Marcus is not sure how much AI's reliance on handcrafted hacks has changed since the late 1990s, except that hacks once written by graduate students are now farmed out to sweatshops as a new form of handcrafting.
No horizon Evidence as of
Limitation: The contemporary contractor report does not establish working conditions, while the retrospective allegations concern an earlier employment period [e1, e2].
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“Turns out you don’t need to solve them to make beautiful pictures, or decent if error-filled conversation, but anyone who thinks we can get to AGI without solving abstraction, compositionality, and the type-token distinction is kidding themselves.”
Our reading: AGI will not be reached unless abstraction, compositionality, and the type-token distinction are solved, and anyone who thinks otherwise is kidding themselves.
No deadline stated Evidence as of
Only if: if abstraction, compositionality, and the type-token distinction are not solved
Assessment: UnresolvedSaid
Gary Marcus Observation
“The measures are new, the model is new, but the stumbling block remains the same.”
Our reading: In the compositionality study he cites, the measures and the model are new, but the stumbling block is the same one neural networks faced in 1993.
No horizon Evidence as of
Limitation: The target period must be refined to the study's actual research period once identified.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“And I still attribute those problems to a lack of world models that represent the ways in which things unfold over time. I still see this as the central flaw in large language models.”
Our reading: The dopey outputs of GPT-style LLMs are caused by a lack of world models of how things unfold over time, which is the central flaw of LLMs.
No horizon Evidence as of
Limitation: Task-specific improvements do not establish a general causal ranking.
Source: Gaslighting and reality in AI