Claims checked against evidence by Gary Marcus, page 6 of 15
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Claims checked against evidence by Gary Marcus (1403)
Each of these claims was checked against dated evidence.
Page 6 of 15: claims 501 to 600 of 1403, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: ContradictedSaid
Causal claim
“And they have such a superficial understanding of their own output that they can’t begin to fact check it.”
Our reading: LLMs have such a superficial understanding of their own output that they can't even begin to fact-check it.
No horizon Evidence as of
Limitation: The verdict turns on the stronger inability claim. A claim of unreliable self-checking would receive a different assessment.
Assessment: UnresolvedSaid
Observation
“And it would be absurd to say that o3 can do all tasks that humans can do in 1.7 hours or that GPT-3 could do anything that a person can do in 15 seconds”
Our reading: o3 cannot do all tasks humans can do in 1.7 hours, and GPT-3 could not do everything a person can do in 15 seconds.
No horizon Evidence as of
Limitation: Unresolved reflects an evidence gap, not evidence that either model has universal capability.
Source: The latest AI scaling graph - and why it hardly makes sense
Assessment: MixedSaid
Observation
“METR’s blog and its tweets (perhaps written by publicists or by generative AI rather than by scientists), however, dropped the careful qualifications and nuance of the technical paper and made claims that go far beyond anything that the study actually supports.”
Our reading: METR's blog and tweets made claims far beyond what its study supports, dropping the technical paper's qualifications.
No horizon Evidence as of
Limitation: This assesses the identified announcements, not every METR tweet.
Source: The latest AI scaling graph - and why it hardly makes sense
Assessment: SupportedSaid
Observation
“Computers can answer some questions reliably now, for example, and some not, and the graph tells us nothing about which is which or when any specific question will be solved.”
Our reading: The METR graph tells us nothing about which questions computers can reliably answer or when specific questions will be solved.
No horizon Evidence as of
Limitation: The outcome depends on reading the claim as about specific questions.
Source: The latest AI scaling graph - and why it hardly makes sense
Assessment: UnresolvedSaid
Causal claim
“Claude does what LaMDA does better because it has more data, but I don’t see any really argument that Claude is any more sentient than a web browser.”
Our reading: Claude does what LaMDA does better because it has more data, but Marcus says he does not see any real argument that Claude is any more sentient than a web browser.
No horizon Evidence as of
Limitation: Private training information limits causal assessment.
Source: New adventures in AI hype: “Our language models are so ‘conscious’ we need to give them rights”
Assessment: SupportedSaid
Observation
“Anthropic is a business (which incidentally neglects to respect the rights of artist and writers who work they nick).”
Our reading: Anthropic fails to respect the rights of the artists and writers whose work it takes.
No horizon Evidence as of
Limitation: Support concerns documented book acquisition, not every creative work or every use.
Source: New adventures in AI hype: “Our language models are so ‘conscious’ we need to give them rights”
Assessment: UnresolvedSaid
Observation
“What these systems do, no more and no less, is to put together sequences of words, but without any coherent understanding of the world behind them, like foreign language Scrabble players who use English words as point-scoring tools, without any clue about what that mean.”
Our reading: LLMs only string together sequences of words and have no coherent understanding of the world behind them.
No horizon Evidence as of
Limitation: The positive mechanistic evidence comes from the model developer and has acknowledged methodological limits.
Source: New adventures in AI hype: “Our language models are so ‘conscious’ we need to give them rights”
Assessment: SupportedSaid
Observation
“It certainly took zero responsibility for what appears to be an obvious infringement.”
Our reading: Sam Altman's response at TED took no responsibility for what appears to be an obvious copyright infringement by an OpenAI system.
No horizon Evidence as of
Limitation: The finding concerns the content of the reply, not the legality of the image.
Source: Sam Altman’s attitude problem
Assessment: UnresolvedSaid
Observation
“NVidia is down significantly more this year (23.4%) than the S&P (11.5%).”
Our reading: NVidia's stock is down 23.4% in 2025 so far, significantly more than the S&P's 11.5% decline.
No horizon Evidence as of
Limitation: Primary historical records sought were inaccessible or returned the wrong period.
Source: Poor ROI for GenAI
Assessment: SupportedSaid
Causal claim
“GenAI is pitched as (near) AGI, but reliability problems persist, limiting how well it can be used in practice.”
Our reading: Although GenAI is marketed as near-AGI, its reliability problems persist and limit how well it can be used in practice.
No horizon Evidence as of
Limitation: The CEO's statement does not establish that existing products were AGI or that every supplier used the same framing.
Source: Poor ROI for GenAI
Assessment: SupportedSaid
Observation
“My sources assure me that this is NOT a clerical error and that Papan’s office confirms that the change is real.”
Our reading: The change replacing AB-501's content with unrelated provisions is real and not a clerical error, according to the author's sources.
No horizon Evidence as of
Limitation: The evidence does not identify Marcus's private sources or verify his communications with them.
Assessment: UnresolvedSaid
Observation
“Real world customers, many of whom are disappointed, are rarely mentioned.”
Our reading: Real-world customers, many of whom are disappointed, are rarely mentioned in the pundits' discussion of AI progress.
No horizon Evidence as of
Limitation: The survey was commissioned by an AI vendor and measures adoption disappointment, which can include organizational problems as well as product shortcomings.
Source: Deep Learning, Deep Scandal
Assessment: SupportedSaid
Observation
“First, the AI community is in fact pretty disappointed with LLama 4’s performance:”
Our reading: The AI community is disappointed with Llama 4's performance.
No horizon Evidence as of
Limitation: Public commentary is self-selected.
Source: Deep Learning, Deep Scandal
Assessment: ContradictedSaid
Observation
“If the juicing of benchmarks and “blending” of test data is the primary scandal, there is a secondary scandal too: a bunch of well-paid pundits, such as Roose, Newton, and Cowen, are failing to reckon with the repeated failures of massive companies to build GPT-5.”
Our reading: A number of well-paid pundits, such as Roose, Newton and Cowen, are failing to reckon with the repeated failures of massive companies to build GPT-5, which Marcus calls a secondary scandal next to benchmark juicing.
No horizon Evidence as of
Limitation: The result depends on reading reckoning as substantive consideration, rather than agreement about how much weight setbacks deserve.
Source: Deep Learning, Deep Scandal
Assessment: UnresolvedSaid
Observation
“Deep learning is indeed finally hitting a wall, in the sense of reaching a point of diminishing results.”
Our reading: Deep learning is hitting a wall in the sense of reaching diminishing returns.
No horizon Evidence as of
Limitation: METR evaluated an earlier GPT-4.5 checkpoint in a short, preliminary evaluation.
Source: Deep Learning, Deep Scandal
Assessment: SupportedSaid
Observation
“models still don’t reason reliably (as anticipated in What to Expect When Your Expecting GPT-4)”
Our reading: LLMs still do not reason reliably as of April 2025.
No horizon Evidence as of
Limitation: The cited versions are preprints.
Source: Scaling is over, the bubble may be deflating, LLMs still can’t reason, and you can’t trust Sam
Assessment: ContradictedSaid
Observation
“Scaling has run out (like I said it would)”
Our reading: Scaling of LLMs has run out as of April 2025.
No horizon Evidence as of
Limitation: The assessment interprets 'run out' as cessation of meaningful capability gains, rather than poor economic returns.
Source: Scaling is over, the bubble may be deflating, LLMs still can’t reason, and you can’t trust Sam
Assessment: UnresolvedSaid
Observation
“An LLM just spits back (sometimes in transmogrified form) what it has read.”
Our reading: An LLM just spits back what it has read, sometimes in transmogrified form.
No horizon Evidence as of
Limitation: Unresolved does not mean that the evidence favors the claim.
Assessment: UnresolvedSaid
Observation
“Humans are (usually, if well-motivated) smart enough to sort things like jokes and fiction from reality, but current AI doesn’t have a clue.”
Our reading: Current AI systems cannot tell jokes and fiction apart from reality, while well-motivated humans usually can.
No horizon Evidence as of
Limitation: The cited studies do not establish either representative failures or a usable ability on the specified fiction-versus-fact task during the assessment period.
Assessment: SupportedSaid
Observation
“In Taming Silicon Valley, I warned that LLMs would be used for dumb things that would affect lots of people.”
Our reading: Marcus says that in Taming Silicon Valley he warned that LLMs would be used for dumb things that would affect lots of people, and he presents the tariff formula, which he says was probably recommended by an LLM, as confirming that warning.
No horizon Evidence as of
Limitation: The complete book was not inspected; the assessment rests on an identified excerpt and an independent review.
Assessment: UnresolvedSaid
Observation
“I do genuinely believe today’s Turing Test takers are better AI systems than an earlier generation. I hardly think that means anything is “over”.”
Our reading: Today's Turing Test-taking AI systems are better than the earlier generation, but that does not mean anything is 'over'.
No horizon Evidence as of
Limitation: This is an evidence gap, not evidence that contemporary systems lack broader improvements.
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: UnresolvedSaid
Conditional forecast
“But I am willing to wager that if someone put in that kind of effort and carefully built out the test, no current system would come close to human performance.”
Our reading: If a careful comprehension challenge test were built, no current AI system would come close to human performance on it.
No deadline stated Evidence as of
Only if: if someone carefully built out the comprehension challenge test
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: SupportedSaid
Causal claim
“To do well the system had to be coached to adopt the persona of “a young person who is introverted, knowledgeable about internet culture, and uses slang”; it’s still in no small part about the tricks.”
Our reading: To do well in the new study, the system had to be coached to adopt the persona of a young, introverted, slang-using person knowledgeable about internet culture, so its success is still in no small part about tricks.
No horizon Evidence as of
Limitation: The comparison does not establish that this particular persona was necessary among all possible prompts [e1].
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: UnresolvedSaid
Observation
“Unfortunately, nobody has ever fully implemented the comprehension challenge test that I proposed.”
Our reading: Nobody has fully implemented Marcus's proposed comprehension challenge test.
No horizon Evidence as of
Limitation: The boundary between a related benchmark and full implementation is not formally defined.
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: UnresolvedSaid
Forecast
“Agents will be a disappointment, precisely because of all of the ongoing reliability and hallucination issues.”
Our reading: AI agents will be a disappointment because of ongoing reliability and hallucination problems.
No deadline stated Evidence as of
Limitation: The evidence mixes enterprise decision-makers, agent developers and deployed-system practitioners.
Assessment: SupportedSaid
Observation
“Especially since DeepSeek there has been a massive price war.”
Our reading: There has been a massive LLM price war, especially since DeepSeek.
No horizon Evidence as of
Limitation: Historical primary announcements were not recovered for the decisive reports; reputable contemporaneous Reuters reporting is used.
Assessment: UnresolvedSaid
Forecast
“For most GenAI model building companies, profits will be meager.”
Our reading: Most GenAI model-building companies will have meager or no profits.
No deadline stated Evidence as of
Limitation: The evidence concerns one company's projected losses before the claim was published, not comparable realized results for a defined population [e1].
Assessment: UnresolvedSaid
Observation
“This And the promised major jumps in economic productivity simply haven’t emerged.”
Our reading: The promised major gains in economic productivity from GenAI have not materialized.
No horizon Evidence as of
Limitation: The US-focused evidence does not establish a worldwide result or isolate GenAI's aggregate contribution in the target quarter [e1, e2].
Assessment: UnresolvedSaid
Observation
“(Or nil; most are losing money; few have made any profits at all).”
Our reading: Most GenAI model-building companies are losing money and few have made any profit.
No horizon Evidence as of
Limitation: Anthropic's historical results were disclosed after the post but before the evaluation date.
Assessment: UnresolvedSaid
Forecast
“The end game here is not that LLMs will vanish, but that LLMs will be virtually free.”
Our reading: LLMs will not vanish but will ultimately become virtually free.
No deadline stated Evidence as of
Limitation: Consumer chat, API inference and downloadable weights have different access costs.
Assessment: SupportedSaid
Observation
“GenAI is an approach that is enormously popular, but one that is neither reliable nor particularly well-grounded in truth.”
Our reading: GenAI is an enormously popular approach, but one that is neither reliable nor particularly well-grounded in truth.
No horizon Evidence as of
Limitation: The adoption figure is company-reported through a news interview, not independently audited.
Assessment: UnresolvedSaid
Observation
“User disillusionment continues to grow.”
Our reading: User disillusionment with generative AI is continuing to grow.
No horizon Evidence as of
Limitation: The satisfaction comparison covers shopping-related use in 2024, not the target period in early 2025 [e1].
Assessment: UnresolvedSaid
Observation
“NVidia, the largest company actually making serious profits from AI, is down 17.75% for the year, and (after a meteoric rise) clearly facing headwinds.”
Our reading: Nvidia, the largest company making serious AI profits, is down 17.75% year-to-date in 2025 and clearly facing headwinds.
No horizon Evidence as of
Limitation: Several promising market-data records could not be read with the required tool.
Assessment: UnresolvedSaid
Forecast
“GenAI is never going to disappear.”
Our reading: Generative AI will never disappear.
No deadline stated Evidence as of
Limitation: The survey covers US Gen Z respondents, not all populations or deployments [e1].
Assessment: SupportedSaid
Observation
“Many other authors are discovering the same thing; social media is full of reports like mine.”
Our reading: Many other authors are discovering that Meta used their works from the pirated database.
No horizon Evidence as of
Limitation: The evidence establishes widespread reports qualitatively, not a measured share of social media.
Source: Meta pirated at least 101 of my books and articles, and tens of millions of others
Assessment: SupportedSaid
Observation
“He worked hard on every one of those books and articles and does not appreciate them being used without permission by AI systems with an established tendency towards regurgitation.”
Our reading: AI systems such as Meta's have an established tendency to regurgitate their training data.
No horizon Evidence as of
Limitation: The studies do not establish how often reproduction occurs in representative everyday conversations or whether Marcus's particular works were reproduced [e1, e2].
Source: Meta pirated at least 101 of my books and articles, and tens of millions of others
Assessment: SupportedSaid
Observation
“I don’t want spoil all the fun, but … things didn’t go well:”
Our reading: In The Guardian's test, people using AI tools without coding knowledge did not succeed well at building a Pac-Man clone.
No horizon Evidence as of
Limitation: The report does not establish universal failure or a population failure rate [e1, e2].
Source: AI Coding Fantasy meets Pac-Man
Assessment: SupportedSaid
Forecast
“To be sure, most or all trained coders will (if they are not already) soon use AI as a part of their workflow.”
Our reading: Most or all trained coders will soon use AI as part of their workflow, if they are not already.
Settled before its deadline Evidence as of
Limitation: These are self-reported surveys, not a census.
Source: AI Coding Fantasy meets Pac-Man
Assessment: UnresolvedSaid
Forecast
“AI will be a tool to help people write code, just as spell-checkers are a tool to help authors write articles and novel, but AI will not soon replace people who understand how to conceive of, write, and debug code.”
Our reading: AI will serve as a tool to help people write code but will not soon replace people who know how to conceive of, write, and debug code.
No deadline stated Evidence as of
Limitation: The available employment research does not directly measure responsibility for conceiving and debugging software.
Source: AI Coding Fantasy meets Pac-Man
Assessment: MixedSaid
Causal claim
“As a fantastic, souped-up version of autocomplete, AI coding tools somewhat increase productivity.”
Our reading: AI coding tools, acting as a fantastic, souped-up version of autocomplete, somewhat increase productivity.
No horizon Evidence as of
Limitation: Neither study establishes an industry-wide average effect or isolates autocomplete from other assistance features [e1, e2].
Source: AI Coding Fantasy meets Pac-Man
Assessment: UnresolvedSaid
Observation
“And they don’t magically resolve the challenges of maintaining complex code bases over time.”
Our reading: Current AI coding tools do not resolve the challenges of maintaining complex code bases over time.
No horizon Evidence as of
Limitation: The study does not establish whether its findings apply to target-period tools [e1].
Source: AI Coding Fantasy meets Pac-Man
Assessment: UnresolvedSaid
Forecast
“The idea that coders (and more generally, software architects) are on their way out is absurd.”
Our reading: Coders and software architects are not on their way out due to AI.
No deadline stated Evidence as of
Limitation: The ordinary occupational-obsolescence reading differs from a claim that employment will never decline.
Source: AI Coding Fantasy meets Pac-Man
Assessment: UnresolvedSaid
Observation
“A few days ago he triumphantly claimed (without evidence) that nearly all code would be written by AI within the next year:”
Our reading: Dario Amodei's claim that nearly all code would be written by AI within the next year was made without evidence.
No horizon Evidence as of
Limitation: The readable transcription is unofficial.
Source: AI Coding Fantasy meets Pac-Man
Assessment: SupportedSaid
Observation
“Musk’s robotaxi promise hasn’t fared much better.”
Our reading: Elon Musk's 2019 promise of over a million Tesla robotaxis has largely failed to come true.
No horizon Evidence as of
Limitation: The filing establishes launch status rather than an exact worldwide robotaxi count.
Assessment: UnresolvedSaid
Observation
“That is not going to magically change next year, at a time when AI still can’t reliably solving river-crossing problems.”
Our reading: Current AI cannot reliably solve river-crossing puzzles.
No horizon Evidence as of
Limitation: Unresolved does not mean the models were reliable.
Assessment: SupportedSaid
Observation
“Even in the best work, like Google’s new AI co-scientist, humans scientists are still choosing the research goals.”
Our reading: Even in the best AI-for-science work, such as Google's AI co-scientist, human scientists still choose the research goals.
No horizon Evidence as of
Limitation: Support is limited to the concrete named example.
Assessment: UnresolvedSaid
Observation
“Nobel-calibre science requires sound logic, reliable causal reasoning, and originality, and none of those are firmly in the grasp of current AI.”
Our reading: Current AI does not firmly have the sound logic, reliable causal reasoning, and originality that Nobel-calibre science requires.
No horizon Evidence as of
Limitation: The two benchmark results are preprints and concern particular tasks and model samples.
Assessment: SupportedSaid
Observation
“Hallucinations haven’t solved, despite constant assurances.”
Our reading: Hallucinations have not been solved despite constant assurances that they would be.
No horizon Evidence as of
Limitation: PersonQA is a company-reported, task-specific evaluation; its rates do not establish rates across everyday use.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Observation
“• AGI should have a reliable ability to understand complex syntactic and semantic structure. (Failings here are also often reflected in quirky errors.”
Our reading: Current AI systems lack a reliable ability to understand complex syntactic and semantic structure, and these failings are often reflected in quirky errors.
No horizon Evidence as of
Limitation: This is an evidence gap, not a finding that the alleged errors do not occur.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Observation
“Especially when we just saw GPT-5 (ultimately relabeled as GPT 4.5) fail despite more than two years of hard and expensive work.”
Our reading: OpenAI's planned GPT-5, relabeled GPT-4.5, failed despite more than two years of hard and expensive work.
No horizon Evidence as of
Limitation: Product improvements can coexist with missed internal expectations.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Observation
“Hallucinations simply haven’t gone away, even in the latest models.”
Our reading: Hallucinations have not gone away, even in the latest models.
No horizon Evidence as of
Limitation: PersonQA concerns factual questions about people; its rate is not an estimate for all uses.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Observation
“• AGI should have reliable ability to reason over abstractions. (Ongoing unpredictable errors in LLMs reflect this.)”
Our reading: LLMs still lack a reliable ability to reason over abstractions, as their ongoing unpredictable errors show.
No horizon Evidence as of
Limitation: The February studies are preprints and do not provide complete execution timestamps.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Causal claim
“But it is not just that I think Klein is wrong, or that he did a disservice to the other side of the argument, but that words like his are motivating a lot of dodgy bets being placed on the imminent AGI assumption — in ways that may not make sense.”
Our reading: Claims like Klein's that AGI is imminent are motivating many dodgy bets that may not make sense.
No horizon Evidence as of
Limitation: Public explanations may omit internal motives.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Observation
“What that means is that a quarter century has passed without a principled solution to even one of those problems.”
Our reading: In the quarter century since 2001, none of the listed AGI prerequisites (generalization, abstract reasoning, tracking individuals, understanding structure) has received a principled solution.
No horizon Evidence as of
Limitation: A narrow successful algorithm would not refute the contextual claim about general capabilities.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Causal claim
“(My view, as I explained in Taming Silicon Valley, is that industry has a vested interest in making people believe that AGI is imminent — how else to justify the lavish valuations - and that they have fooled a lot of people in government who aren’t technical experts into believing that AGI is closer than it really is).”
Our reading: The AI industry has a vested interest in claiming AGI is imminent to justify lavish valuations, and has misled many non-technical government officials into thinking AGI is closer than it is.
No horizon Evidence as of
Limitation: One company's announcement cannot establish an industry-wide valuation mechanism.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Observation
“The fact that Grok 3 was 15x the size of Grok 2, and also generally viewed as disappointing.”
Our reading: Grok 3 was 15 times the size of Grok 2 and was generally viewed as disappointing.
No horizon Evidence as of
Limitation: The scale judgment depends on what size means.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Observation
“Most parts of the US still don’t have functioning driverless cars systems, for example, even after over a decade of lavish promises.”
Our reading: Most parts of the US still lack functioning driverless car systems after more than a decade of lavish promises.
No horizon Evidence as of
Limitation: Geographic coverage is assessed qualitatively rather than through a national area or population calculation.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Observation
“But it is absolutely absurd to pretend “AGI imminent” hypothesis isn’t getting play in high places.”
Our reading: The hypothesis that AGI is imminent is getting play in high places, and it is absurd to pretend otherwise.
No horizon Evidence as of
Limitation: This supports the general observation, not every example or institutional explanation in the surrounding post.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: SupportedSaid
Observation
“Every red cell in this graph (e.g, multiplying 20 digit numbers by 19 digit numbers2) is failure to generalize, even after immense amounts of training data:”
Our reading: LLMs including o1-mini fail to generalize on large multi-digit multiplication despite immense training data.
No horizon Evidence as of
Limitation: The retrieved text does not permit a cell-by-cell audit, including the specific 20-by-19-digit example.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Observation
“The fact that GPT 4.5 received very mixed reviews — despite years of labor and billions investment — and could in no reasonable way be considered a major (rather than minor, incremental) step forwards towards AGI.”
Our reading: GPT-4.5 received very mixed reviews and is at most a minor, incremental step toward AGI, despite years of work and billions invested.
No horizon Evidence as of
Limitation: Company benchmarks and limited pre-release evaluations do not exhaust a model's capabilities.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Forecast
“And these complex systems, built on the unreliable foundations of LLMs, are going to prove to be massively difficult to debug.”
Our reading: Complex systems built on LLMs, such as deep research agents, will prove massively difficult to debug.
No deadline stated Evidence as of
Limitation: The production account is a company report without quantified debugging effort.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Forecast
“But there’s also no reason to think that any of them are logically insolvable in principle. So we will get there someday.”
Our reading: Because there is no reason to think any of the remaining problems are logically unsolvable in principle, AGI will be achieved someday.
No deadline stated Evidence as of
Limitation: The search cannot establish that every possible demonstration has been examined.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Forecast
“AI might replace average people in some tasks soon; replacing the best humans in a few years? Not going to happen.”
Our reading: AI will not replace the best humans, such as top scientists, within a few years, though it might replace average people in some tasks soon.
Not yet due Evidence as of
Limitation: The exact deadline and meaning of replacement are interpretive.
Source: Ezra Klein’s new take on AGI – and why I think it’s probably wrong
Assessment: UnresolvedSaid
Forecast
“I can’t imagine that noncoders are really going to get those systems to work well enough for prime time (except perhaps in very very narrow use cases), let alone know what do when the back end internet calls break.”
Our reading: Marcus says he can't imagine noncoders getting systems built by automated code tools to work well enough for prime time, except perhaps in very narrow use cases.
No deadline stated Evidence as of
Limitation: Both sources are company accounts without independent operational verification.
Assessment: UnresolvedSaid
Observation
“They never let you know that a significant number of real world coders are deeply frustrated with automated coding tools like Cursor.AI (which is in part powered by Anthropic’s Claude):”
Our reading: A significant number of real-world coders are deeply frustrated with automated coding tools such as Cursor, which is partly powered by Anthropic's Claude.
No horizon Evidence as of
Limitation: Significant number and deeply frustrated remain qualitative.
Assessment: UnresolvedSaid
Observation
“Nine years later and hundreds of startups later, my understanding is that not one radiologist has been replaced, and many parts of the world actually have a shortage of radiologists.”
Our reading: Marcus's understanding is that nine years and hundreds of startups after Hinton's 2016 prediction, not one radiologist has been replaced, and many parts of the world have a shortage of radiologists.
No horizon Evidence as of
Limitation: The outcome depends on reading replacement as job displacement rather than substitution on individual tasks.
Assessment: UnresolvedSaid
Forecast
“People are increasingly going to discourage American children from learning to code based on naive enthusiasm from people like Roose.”
Our reading: Hype about AI coding will increasingly lead people to discourage American children from learning to code.
No deadline stated Evidence as of
Limitation: Public attitudes toward coding are not the same as actual advice to children.
Assessment: UnresolvedSaid
Forecast
“At least for another decade or two and probably more, big projects are always going to require software architects who know what they are doing.”
Our reading: For at least another 10 to 20 years, and probably longer, big software projects will require competent human software architects.
Not yet due Evidence as of
Limitation: The distinction between a substantial benchmark implementation and a big software project requires interpretation.
Assessment: UnresolvedSaid
Observation
“it’s often easy to get a solution that is 80% correct, and extremely difficult to get the last 20%. Driverless cars have been like that for decades.”
Our reading: In AI it is often easy to reach 80% correctness and extremely difficult to get the last 20%, as with driverless cars for decades.
No horizon Evidence as of
Limitation: Distance automated is not equivalent to correctness across all driving situations.
Assessment: SupportedSaid
Observation
“A new study of the latest system, GPT 4.5 showed that by on one standardized measure over 1/3 of the outputs are still hallucinations.”
Our reading: On one standardized measure, over a third of GPT-4.5's outputs are hallucinations.
No horizon Evidence as of
Limitation: The exact percentage is supported by Computerworld's report of OpenAI's evaluation, not a readable primary numeric table.
Assessment: SupportedSaid
Observation
“Already, of course, DeepSeek has shown that our once strong lead is diminishing.”
Our reading: DeepSeek has shown that the US's once-strong lead is shrinking.
No horizon Evidence as of
Limitation: The report was published after the claim, although its measurements cover the relevant earlier period.
Assessment: SupportedSaid
Observation
“Here Roose has displayed the cardinal sin of looking only at regurgitation rather than generalization. History has told us over and over that systems can be good at the former and still face enormous challenges at the latter.”
Our reading: AI systems can be good at regurgitating known code while still facing enormous challenges generalizing to new problems.
No horizon Evidence as of
Limitation: Training similarity and robustness failures do not by themselves prove verbatim memorization.
Assessment: SupportedSaid
Observation
“And of course, there is still no GPT-5, even after two years, of hype, bullshit, and cryptic hints.”
Our reading: OpenAI still has not released GPT-5 after two years of hype.
No horizon Evidence as of
Limitation: The assessment concerns public availability, not internal prototypes.
Source: OpenAI, in deep trouble
Assessment: UnresolvedSaid
Observation
“Even Altman himself was uncharacteristically tepid in his product announcement.”
Our reading: Sam Altman was unusually tepid in announcing GPT-4.5, not mentioning AGI.
No horizon Evidence as of
Limitation: The packet provides only one earlier product announcement, which cannot establish Altman's usual tone. [e3]
Assessment: SupportedSaid
Observation
“So now GPT 4.5 has dropped, and — breaking news — I was right. Hallucinations didn’t disappear today, and nor did stupid errors.”
Our reading: GPT-4.5 did not eliminate hallucinations or stupid errors.
No horizon Evidence as of
Limitation: The evidence is developer-reported.
Assessment: UnresolvedSaid
Observation
“Gary Marcus was right all along that pure scaling would eventually hit a wall, or at least reach a point of massively diminishing returns.”
Our reading: Pure scaling of LLMs has hit a wall or at least massively diminishing returns.
No horizon Evidence as of
Limitation: The launch comparisons do not isolate the contribution of pretraining scale or provide a controlled measure of returns to additional resources. [e1]
Assessment: UnresolvedSaid
Observation
“Right now, the math for profitability doesn’t work, influence on productivity of GenAI has been minimal, and prospects for pure scaling aren’t looking great.”
Our reading: The math for generative AI profitability currently doesn't work.
No horizon Evidence as of
Limitation: Available loss reporting is not an industry financial statement.
Source: GPT 4.5 is no GPT-5
Assessment: UnresolvedSaid
Observation
“Right now, the math for profitability doesn’t work, influence on productivity of GenAI has been minimal, and prospects for pure scaling aren’t looking great.”
Our reading: Generative AI's influence on productivity has been minimal.
No horizon Evidence as of
Limitation: The original survey estimates were subsequently revised.
Source: GPT 4.5 is no GPT-5
Assessment: UnresolvedSaid
Observation
“It’s not just OpenAI either; it’s the whole field, as I discussed in my own recent Fortune article:”
Our reading: Diminishing returns from scaling affect the whole AI field, not just OpenAI.
No horizon Evidence as of
Limitation: Important underlying experiments were confidential.
Source: GPT 4.5 is no GPT-5
Assessment: SupportedSaid
Forecast
“Actually, at any moment, we will finally get GPT 4.5, which most certainly wouldn’t live up to expectations around GPT-5 — hence its rather tentative name.”
Our reading: GPT 4.5 will not live up to the expectations that had been set for GPT-5.
Settled before its deadline Evidence as of
Limitation: The decisive measurements are company-reported.
Source: GPT 4.5 is no GPT-5
Assessment: UnresolvedSaid
Observation
“The truth is that scaling just isn’t producing the fruit that it once did.”
Our reading: Scaling is no longer producing the gains it once did.
No horizon Evidence as of
Limitation: The available reporting is more specific to particular projects than the claim.
Source: GPT 4.5 is no GPT-5
Assessment: UnresolvedSaid
Observation
“And for that matter people should be comparing GenAI revenue (on the order of low tens of billions per year, pre price wars) with GenAI expenditures (order half a trillion dollars to date), and doing the math, especially given the price wars induced by DeepSeek.”
Our reading: GenAI revenue is on the order of low tens of billions of dollars per year (before the price wars), while GenAI expenditures are on the order of half a trillion dollars to date.
No horizon Evidence as of
Limitation: The claim combines different periods and incompletely defined industry boundaries.
Source: GPT 4.5 is no GPT-5
Assessment: SupportedSaid
Observation
“In “Deep learning is hitting a wall”, I warned that problems with reasoning and hallucination wouldn’t go away simply as a function of scaling, and they haven’t.”
Our reading: Problems with reasoning and hallucination in deep learning systems have not gone away as a function of scaling.
No horizon Evidence as of
Limitation: Humanity's Last Exam combines knowledge and reasoning; not every wrong answer identifies a reasoning failure.
Source: GPT 4.5 is no GPT-5
Assessment: UnresolvedSaid
Causal claim
“I personally noticed Musk’s shutting out of opposing views when he stopped answering my emails when I warned him in July – correctly - that throwing in with Trump would undermine the cause of AI safety (which for a decade, until a few weeks ago, seemed extremely important to Musk).”
Our reading: Musk's alignment with Trump has undermined the cause of AI safety, as Marcus correctly warned in July 2024.
No horizon Evidence as of
Limitation: No admitted evidence establishes Musk's causal contribution to the federal advanced-AI policy rollback.
Source: Elon Musk's inability to listen to others is torching almost everything he touches
Assessment: SupportedSaid
Observation
“Simply false, in view of the many basic aspects of geometry that AlphaGeometry2 does not cover, as discussed above.”
Our reading: The claim that AlphaGeometry2 'just figured out geometry' is false because it does not cover many basic aspects of geometry.
No horizon Evidence as of
Limitation: Support depends on reading figured out geometry as general mastery.
Source: AlphaGeometry2: Impressive accomplishment, but still a long path ahead
Assessment: UnresolvedSaid
Observation
“Taking this into account, the fraction of problems where a composite AI system could solve the problem starting with its English language statement is at most 60%.”
Our reading: Of the 50 IMO geometry problems from 2000 to 2024, a composite AI system starting from the English problem statements could solve at most 60%.
No horizon Evidence as of
Limitation: Automatic translation success does not itself establish solving success.
Source: AlphaGeometry2: Impressive accomplishment, but still a long path ahead
Assessment: UnresolvedSaid
Observation
“DeepMind didn’t make that claim, and it isn’t true.”
Our reading: The TechCrunch headline's claim about AlphaGeometry2 was not made by DeepMind and is not true.
No horizon Evidence as of
Limitation: The packet contains no quoted DeepMind report or matched performance evidence.
Source: AlphaGeometry2: Impressive accomplishment, but still a long path ahead
Assessment: SupportedSaid
Observation
“Unlike DeepMind’s AlphaZero system which learned to play multiple board games, or OpenAI’s ChatGPT, AlphaGeometry2 is nowhere close to being a single general-purpose learning architecture.”
Our reading: AlphaGeometry2 is a specialized system, nowhere close to a single general-purpose learning architecture.
No horizon Evidence as of
Limitation: The documentation became available after Marcus’s February 17 post.
Source: AlphaGeometry2: Impressive accomplishment, but still a long path ahead
Assessment: UnresolvedSaid
Conditional forecast
“Even if one were to glom together the best general AIs with AlphaGeometry2, the combination would fall far short of the geometric understanding and problem solving ability of high school math student with a talent for math.”
Our reading: Combining the best general AIs with AlphaGeometry2 would still fall far short of a talented high school math student's geometric understanding and problem-solving.
No deadline stated Evidence as of
Only if: if the best general AIs were combined with AlphaGeometry2
Source: AlphaGeometry2: Impressive accomplishment, but still a long path ahead
Assessment: SupportedSaid
Observation
“About year ago the City of New York tried this kind of thing, with less than spectacular results:”
Our reading: About a year ago the City of New York tried this kind of thing (deploying problematic, unreliable AI to the public), with less than spectacular results.
No horizon Evidence as of
Limitation: The documented tests do not establish a representative overall error rate.
Assessment: UnresolvedSaid
Observation
“Over the last couple days I have been asking version 2 to draw images identifying and tagging parts of images. The results are consistently awful:”
Our reading: When Marcus asked Grok 2 over the last couple of days to draw images identifying and tagging parts of images, the results were consistently awful.
No horizon Evidence as of
Limitation: Text extraction cannot establish what the embedded images show.
Assessment: UnresolvedSaid
Forecast
“And it will be the citizens of the United States who bear the consequences, while the guy who stood to get a $400M armored T̶e̶s̶l̶a̶ EV government contract (until the media squawked) stands to collect a big slice of the profits, unemploying thousands of lifelong government employees.”
Our reading: US citizens will bear the consequences, while Musk stands to collect a big slice of the profits, putting thousands of lifelong government employees out of work.
No deadline stated Evidence as of
Limitation: Employment totals combine different reasons for departure and do not establish AI causation.
Assessment: UnresolvedSaid
Observation
“Yet Musk wants to use shove problematic AI, both biased and unreliable, down our throats, immediately, as reported by The New York Times, and noted in this Bluesky comment:”
Our reading: Musk wants to deploy biased and unreliable AI in government immediately.
No horizon Evidence as of
Limitation: The usable evidence is news reporting based on sources describing an internal meeting; no primary meeting record was successfully read.
Assessment: UnresolvedSaid
Forecast
“Soon such unreliable stuff, with butt-covering warning messages that most people will ignore, will be everywhere.”
Our reading: Soon such unreliable AI, with butt-covering warning messages that most people will ignore, will be everywhere.
No deadline stated Evidence as of
Limitation: The search does not establish that qualifying evidence cannot exist.
Assessment: UnresolvedSaid
Forecast
“For nearly three years I have been saying that the pure scaling of LLMs – adding more data and more parameters – would eventually run out, and that it would fail to solve hallucinations and boneheaded errors and that scaling laws were merely empirical generalizations, rather than physical laws.”
Our reading: Marcus restates his long-held prediction that pure scaling of LLMs (adding more data and more parameters) would eventually run out and would fail to solve hallucinations and boneheaded errors, and that scaling laws are merely empirical generalizations rather than physical laws.
No deadline stated Evidence as of
Limitation: Available model comparisons change several factors simultaneously.
Source: Breaking: OpenAI's efforts at pure scaling have hit a wall.
Assessment: SupportedSaid
Forecast
“Second, pure scaling will no longer be the means of attack.”
Our reading: OpenAI will no longer rely on pure scaling as its means of building GPT-5.
Settled before its deadline Evidence as of
Limitation: The disclosure establishes the additional mechanisms, not their precise share of capability gains.
Source: Breaking: OpenAI's efforts at pure scaling have hit a wall.
Assessment: UnresolvedSaid
Observation
“They were overhyping LLMS as AGI, even as problems like hallucinations and boneheaded persisted.”
Our reading: As a premise of Taming Silicon Valley, tech oligarchs were overhyping LLMs as AGI even as problems like hallucinations and boneheaded errors persisted.
No horizon Evidence as of
Limitation: The packet lacks a specific executive AGI claim matched to contemporary evaluations of the corresponding systems and capabilities.
Source: Everything I warned about in Taming Silicon Valley is rapidly becoming our reality
Assessment: UnresolvedSaid
Causal claim
“(Also, having our own massive LLMs will do nothing to stop China from using theirs in authoritarian pursuits.)”
Our reading: US possession of massive LLMs will do nothing to stop China from using its LLMs for authoritarian purposes.
No horizon Evidence as of
Limitation: Public records provide limited visibility into covert deployments and their outcomes.
Source: Everything I warned about in Taming Silicon Valley is rapidly becoming our reality
Assessment: ContradictedSaid
Forecast
“The new administration will do everything in its power to protect businesses, and nothing to protect individuals.”
Our reading: The new administration will do everything in its power to protect businesses, and nothing to protect individuals.
Settled before its deadline Evidence as of
Limitation: The enactment records do not measure enforcement or effectiveness.
Source: Everything I warned about in Taming Silicon Valley is rapidly becoming our reality
Assessment: UnresolvedSaid
Observation
“LLMs, once novel, are largely a commodity.”
Our reading: LLMs have largely become a commodity.
No horizon Evidence as of
Limitation: The survey’s 2024 findings cannot be isolated to the target window [e1].