Claims checked against evidence by Gary Marcus, page 7 of 19
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Claims checked against evidence by Gary Marcus (1815)
Each of these claims was checked against dated evidence.
Page 7 of 19: claims 601 to 700 of 1815, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: UnresolvedSaid
Causal claim
“Their stories about this are, in my view, greatly flawed, but having outside groups with science fiction chops writing stuff like this distracts away from those flaws, and gives more power to the very companies trying hardest to race towards AGI.”
Our reading: OpenAI's and Anthropic's AGI-imminence stories are greatly flawed, and outside scenarios like AI 2027 distract from those flaws and empower those companies.
No horizon Evidence as of
Limitation: The criticism of the company stories is underspecified and partly evaluative.
Assessment: UnresolvedSaid
Observation
“And so far, anyway, the impact on the labor market of Generative AI has been modest; ditto for productivity in any measurable statistics.”
Our reading: So far, generative AI's impact on the labor market and on measurable productivity has been modest.
No horizon Evidence as of
Limitation: Geographic and occupational coverage is limited.
Assessment: UnresolvedSaid
Causal claim
“AI 2027 isn’t slowing them down; it’s putting wind (and money, and political power) in their sails.”
Our reading: AI 2027 is giving AI companies like OpenAI and Anthropic more money and political power rather than slowing them.
No horizon Evidence as of
Limitation: Investor motives and private political discussions may be unavailable.
Assessment: UnresolvedSaid
Observation
“Remember how we were told in 2012 that we would all have driverless cars by 2017? That hasn’t come to pass, either, except in about 10 of the world’s 20,000 cities.”
Our reading: Driverless cars, promised for everyone by 2017, operate in only about 10 of the world's 20,000 cities.
No horizon Evidence as of
Limitation: There is no deduplicated worldwide inventory using consistent passenger-access, supervision, and geographic criteria.
Assessment: UnresolvedSaid
Causal claim
“The all too common strategy of “scare everyone with the thought of superintelligence” is backfiring, because it’s making it seem that superintelligence is inevitable, and unstoppable, and achievable only by the companies we have today.”
Our reading: Scaring people about superintelligence backfires because it makes superintelligence seem inevitable, unstoppable, and achievable only by today's companies.
No horizon Evidence as of
Limitation: The supplied evidence measures stated attitudes rather than observed policy behavior.
Assessment: UnresolvedSaid
Forecast
“It’s also encouraging the world to make short-term choices about AI (e.g., making plans around export controls when China will inevitably catch up) rather than longer-term choices (investing in research to develop safer, more alignable approaches to AI).”
Our reading: China will inevitably catch up in AI despite export controls.
No deadline stated Evidence as of
Limitation: The interpretation of catch up materially affects the result.
Assessment: UnresolvedSaid
Causal claim
“It’s also encouraging the world to make short-term choices about AI (e.g., making plans around export controls when China will inevitably catch up) rather than longer-term choices (investing in research to develop safer, more alignable approaches to AI).”
Our reading: AI 2027 is encouraging the world to make short-term choices about AI, such as planning around export controls when China will inevitably catch up, rather than longer-term choices such as investing in research on safer, more alignable approaches to AI.
No horizon Evidence as of
Limitation: Public announcements may omit influences on internal deliberations.
Assessment: UnresolvedSaid
Causal claim
“The second issue is that by writing scenarios that trade so heavily on China-US conflict, they are feeding the worst fears of hawks, both in the US and China — escalating how much money and power both sides will give to companies racing as fast as possible, and reduce investments in efforts to mitigate the risks of building AI that we are scarcely able to control.”
Our reading: US-China conflict framing in AI scenarios feeds hawks on both sides, increasing money and power for racing AI companies and reducing investment in AI risk mitigation.
No horizon Evidence as of
Limitation: Public attitudes are not equivalent to policy decisions or funding allocations.
Assessment: UnresolvedSaid
Forecast
“Overall, then, the AI 2027 scenario almost certainly underestimates how much time we have to prepare for general intelligence, by years if not decades.”
Our reading: General intelligence will almost certainly arrive years, if not decades, later than the AI 2027 scenario projects.
No deadline stated Evidence as of
Limitation: A precise outcome will depend on which capability milestone counts as general intelligence.
Assessment: SupportedSaid
Observation
“Remember how we were told in 2023 that hallucinations would be solved in a matter of months? They are still here.”
Our reading: Despite 2023 promises that hallucinations would soon be solved, they persist as of May 2025.
No horizon Evidence as of
Limitation: Relies mainly on one developer's internal benchmarks.
Assessment: UnresolvedSaid
Observation
“A big part of the issue— which AI 2027 never faces — is that we simply don’t know how to create synthetic data for many other domains, like what might happen if India and Pakistan go to war using drones or what the effects of 30% tariff might be on the global economy.”
Our reading: We do not know how to create useful synthetic training data for many open-ended, unverifiable domains.
No horizon Evidence as of
Limitation: A single successful forecasting method cannot refute a claim about many other domains.
Assessment: UnresolvedSaid
Causal claim
“Tall tales about the imminence of AGI aren’t slowing down the AI race dynamic that the authors of AI-2027 want to mitigate; they are speeding that very dynamic up.”
Our reading: Stories claiming AGI is imminent are speeding up, not slowing, the AI race dynamic.
No horizon Evidence as of
Limitation: The accessible experimental evidence measures attention, not development decisions.
Assessment: UnresolvedSaid
Observation
“Cell phones and the internet were pretty big, but it’s not obvious their impact was greater than the industrial revolution, and so far AI’s impact has (despite all the PR) been far less than either cell phones or the internet.”
Our reading: So far, AI's impact has been far less than that of cell phones or the internet.
No horizon Evidence as of
Limitation: The evidence provides no common measure of accumulated impact or threshold for 'far less'.
Assessment: ContradictedSaid
Observation
“Absolutely nothing is being done about AI-generated misinformation, political or medical.”
Our reading: Nothing is being done about political or medical AI-generated misinformation.
No horizon Evidence as of
Limitation: The measure targets caller authentication and transmission practices, not every form of misinformation.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: ContradictedSaid
Observation
“Now he is doing everything he can to stop AI regulation of any meaningful sort.”
Our reading: Sam Altman is doing everything he can to stop meaningful AI regulation.
No horizon Evidence as of
Limitation: OpenAI's endorsement establishes an institutional position, not a separate personal endorsement by Altman or his private beliefs.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: ContradictedSaid
Observation
“The shift in norms has extended to our government, too; gone are the days when Washington was afraid of the risks of AI enough to seriously consider doing anything about them.”
Our reading: Washington no longer seriously considers acting on AI risks.
No horizon Evidence as of
Limitation: Targeted measures do not establish comprehensive regulation of frontier AI developers.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: MixedSaid
Observation
“Congress has failed to pass any meaningful AI regulation, and even worse, they are now actively aiming to prevent States — probably our last hope — from passing anything meaningful.”
Our reading: Congress has passed no meaningful AI regulation and is trying to prevent states from passing meaningful AI regulation.
No horizon Evidence as of
Limitation: Congressional passage does not establish enactment or that the bill's duties were already in force.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: SupportedSaid
Observation
“(He also said at the time “we think that content creators, content owners, need to benefit from this technology” but ever since his company has been pushing for free training materials and exemption from copyright laws.)”
Our reading: Since 2023 OpenAI has pushed for free training materials and exemption from copyright law, contrary to Altman's stated support for content creators.
No horizon Evidence as of
Limitation: Fair use is part of copyright law; calling it an exemption from copyright laws is imprecise.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: MixedSaid
Observation
“Government officials aren’t worrying out loud about the risks of AI, anymore. They are downplaying them.”
Our reading: US government officials now downplay rather than voice concern about AI risks.
No horizon Evidence as of
Limitation: The Federal Reserve is part of the federal government but does not speak for the administration.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: SupportedSaid
Observation
“And even as Altman was telling Congress that he supported AI regulation, his company was lobbying the EU to water down its AI act.”
Our reading: While Altman publicly supported AI regulation in 2023, OpenAI lobbied to water down the EU AI Act.
No horizon Evidence as of
Limitation: The temporal reading is material: a requirement for an EU weakening request on the exact hearing date would remain unresolved.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: UnresolvedSaid
Observation
“We have no concretely implemented international AI governance, no national AI agency; we are no longer even positioned well to detect and address AI-escalated cybercrime.”
Our reading: As of May 2025, there is no concretely implemented international AI governance and no US national AI agency, and the US is no longer even well positioned to detect and address AI-escalated cybercrime.
No horizon Evidence as of
Limitation: The OECD article is retrospective and carries an authors' views disclaimer.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: UnresolvedSaid
Observation
“By many accounts, AI-fueled scams have exploded, too, and again there is no coherent federal response.”
Our reading: By many accounts, AI-fueled scams have exploded, and there is no coherent US federal response to them.
No horizon Evidence as of
Limitation: Vendor detection data can change with customer coverage and detection methods.
Source: Two Years Ago Today in AI History: The Tale of An About-face in AI Regulation
Assessment: SupportedSaid
Observation
“And then his AI, Grok, would not shut up about white genocide, even when it is completely irrelevant.”
Our reading: In May 2025 Grok repeatedly brought up 'white genocide' in South Africa even in replies where it was completely irrelevant.
No horizon Evidence as of
Limitation: Original exchange archives were inaccessible.
Source: Has Grok lost its mind and mind-melded with its owner?
Assessment: SupportedSaid
Forecast
“But the writing is on the wall. Sooner or later, powerful people are going to use LLMs to shape your ideas.”
Our reading: Sooner or later, powerful people are going to use LLMs to shape people's ideas.
Settled before its deadline Evidence as of
Limitation: The result depends on including state-directed organizations and distributed generated messages within the forecast.
Source: Has Grok lost its mind and mind-melded with its owner?
Assessment: UnresolvedSaid
Causal claim
“Because both countries are both building essentially the same kind of models with the same kinds of techniques using the same kinds of data, the results from the two nations are essentially converging on the same outcomes.”
Our reading: US and Chinese AI results are converging because both use the same kinds of models, techniques, and data.
No horizon Evidence as of
Limitation: A supported assessment of benchmark convergence alone would drop the claim's causal commitment.
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: SupportedSaid
Observation
“But the reality is that self-regulation has repeatedly failed the public, and the absence of oversight would only invite more industry lobbying to maintain weak accountability.”
Our reading: Tech industry self-regulation has repeatedly failed the public.
No horizon Evidence as of
Limitation: The examples concern privacy and online platforms; they do not measure every area of AI safety.
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Causal claim
“With no regulatory pressure, tech companies would have little incentive to prioritize safety, transparency, or ethical design; any costs to society would be borne by society.”
Our reading: With no regulatory pressure, tech companies would have little incentive to prioritize safety, transparency, or ethical design, and any costs to society would be borne by society.
No horizon Evidence as of
Only if: with no regulatory pressure
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Conditional forecast
“First, if current trends continue, the “race” with China will not end in victory, for either side.”
Our reading: If current trends continue, neither the US nor China will win the AI race.
No deadline stated Evidence as of
Only if: if current trends continue
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Observation
“So-called leaderboards are no longer dominated by any one country.”
Our reading: AI leaderboards are no longer dominated by any single country.
No horizon Evidence as of
Limitation: Several categories from one benchmark operator are not independent measures.
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: SupportedSaid
Observation
“In 2024 alone, 31 states, Puerto Rico, and the Virgin Islands enacted AI-related legislation or resolutions, and at least 27 states passed deepfake laws.”
Our reading: In 2024 alone, 31 states, Puerto Rico, and the Virgin Islands enacted AI-related legislation or resolutions, and at least 27 states passed deepfake laws.
No horizon Evidence as of
Limitation: The source is an advocacy letter by participating legislators, not an independent audit.
Source: When it Comes to AI Policy, Congress Shouldn’t Cut States off at the Knees
Assessment: UnresolvedSaid
Forecast
“With enough money you can have TAs and smaller sections of say 20 students each, taught by teaching assistants, but with university budgets being slashed, few universities will have enough money to make that work.”
Our reading: With university budgets being cut, few universities will have enough money to replace term papers with small TA-led sections.
No deadline stated Evidence as of
Limitation: US universities are an interpretive population choice.
Source: ”Everyone is cheating their way through college” with GenAI. Who should bear the costs?
Assessment: UnresolvedSaid
Causal claim
“GenAI has nuked all of this.”
Our reading: GenAI has destroyed term papers' role in giving writing practice, prompting reflection and evaluating students at scale.
No horizon Evidence as of
Limitation: The strongest negative experiment examines take-home examinations rather than term papers.
Source: ”Everyone is cheating their way through college” with GenAI. Who should bear the costs?
Assessment: UnresolvedSaid
Causal claim
“The creators and distributors of GenAI, who have brought this huge problem to society?”
Our reading: The creators and distributors of GenAI have brought this huge problem, GenAI undermining college education and evaluation, to society.
No horizon Evidence as of
Limitation: The strongest direct experiment covers five psychology modules at one UK university.
Source: ”Everyone is cheating their way through college” with GenAI. Who should bear the costs?
Assessment: UnresolvedSaid
Observation
“If I can be blunt, it is an absolute embarrassment that a technology that has collectively cost about half a trillion dollars can’t do something as basic as (reliably) check its output against wikipedia or a CNN article that is handed on a silver plattter.”
Our reading: LLM technology, having cost about half a trillion dollars, cannot reliably check its output against Wikipedia or a supplied article.
No horizon Evidence as of
Limitation: The money component has an unknown accounting basis.
Assessment: SupportedSaid
Observation
“But the problem is not going away; OpenAI’s recent o3 actually hallucinates more than some its predecesssors.”
Our reading: The hallucination problem is not going away; OpenAI's o3 hallucinates more than some of its predecessors.
No horizon Evidence as of
Limitation: These are provider-reported results on two factual-question benchmarks.
Assessment: UnresolvedSaid
Observation
“The extra embarrassing part about all of this for GenAI is that almost of the above information could have been found quickly and easily with a two second search for his wikipedia page, and most of it could be found in the first screenful at that.”
Our reading: Almost all of the information about Harry Shearer that the AI biography got wrong or left out could have been found quickly and easily with a two-second search for his Wikipedia page, and most of it could be found in the first screenful.
No horizon Evidence as of
Limitation: A text introduction is not equivalent to a screenful.
Assessment: UnresolvedSaid
Causal claim
“Because LLMS statistically mimic the language people have used, they often fool people into thinking that they operate like people.”
Our reading: Because LLMs statistically mimic human language, they often fool people into thinking they operate like people.
No horizon Evidence as of
Limitation: A broader reading limited to apparent humanlike behavior would receive stronger support.
Assessment: ContradictedSaid
Observation
“But they don’t operate like people. They don’t, for example, ever fact check (as humans sometimes, when well motivated, do). They mimic the kinds of things of people say in various contexts. And that’s essentially all they do.”
Our reading: LLMs do not operate like people and never fact check; they essentially only mimic what people say in contexts.
No horizon Evidence as of
Limitation: The counterexample uses an engineered workflow with retrieval and prompting.
Assessment: UnresolvedSaid
Causal claim
“(Those systems also can’t reliably stick to the rules of chess despite having them – and millions of games – in their database, a manifestation of the related problem of extracting statistical tendencies without every fully deriving and apprehending the correct abstractions).”
Our reading: LLMs can't reliably stick to the rules of chess despite having the rules and millions of games in their data, and this is a manifestation of the related problem of extracting statistical tendencies without ever fully deriving and grasping the correct abstractions.
No horizon Evidence as of
Limitation: Puzzle results, full-game results, and playing strength are not interchangeable.
Assessment: UnresolvedSaid
Causal claim
“The chronic problem with creating fake citations in research papers and faked cases in legal briefs is a manifestation of the same problem; LLMs correctly “model” the structure of academic references, but often make up titles, page numbers, journals and so on — once again failing to sanity check their outputs against information (in this case lists of references) that are readily found on the internet.”
Our reading: Fake citations and faked legal cases from LLMs stem from the same problem of mimicking structure without sanity-checking against available information.
No horizon Evidence as of
Limitation: Successful mitigation is evidence about contributors, but a combined system change does not isolate every causal component.
Assessment: UnresolvedSaid
Causal claim
“To a next-token predictor, a phrase like Harry Shearer lives in a particular part of a multidimensional space. Words in that space are often followed by words like “British actor”. So out comes a hallucination.”
Our reading: The AI's hallucination that Harry Shearer is a 'British actor' came about because, to a next-token predictor, a phrase like 'Harry Shearer' sits in a part of a multidimensional space where words are often followed by words like 'British actor'.
No horizon Evidence as of
Limitation: Public searches cannot establish that no relevant internal investigation exists.
Assessment: UnresolvedSaid
Observation
“They don’t have a database of records like any proper business would (which would be a strong basis to solve the problem); and they don’t have what people like Yann LeCun or Judea Pearl or I would call a world model.”
Our reading: LLMs don't have a database of records like any proper business would, and they don't have what people like Yann LeCun, Judea Pearl, or Marcus would call a world model.
No horizon Evidence as of
Limitation: One documented architecture cannot establish a claim about all LLMs.
Assessment: UnresolvedSaid
Forecast
“We will eventually get to something better, but continuing to put all our eggs in the Henrietta’s LLM basket is absurd to the point of being delusional.”
Our reading: AI will eventually get to something better than LLMs.
No deadline stated Evidence as of
Limitation: The result depends on how broadly 'something better' is read.
Assessment: ContradictedSaid
Causal claim
“And they have such a superficial understanding of their own output that they can’t begin to fact check it.”
Our reading: LLMs have such a superficial understanding of their own output that they can't even begin to fact-check it.
No horizon Evidence as of
Limitation: The verdict turns on the stronger inability claim. A claim of unreliable self-checking would receive a different assessment.
Assessment: UnresolvedSaid
Observation
“And it would be absurd to say that o3 can do all tasks that humans can do in 1.7 hours or that GPT-3 could do anything that a person can do in 15 seconds”
Our reading: o3 cannot do all tasks humans can do in 1.7 hours, and GPT-3 could not do everything a person can do in 15 seconds.
No horizon Evidence as of
Limitation: Unresolved reflects an evidence gap, not evidence that either model has universal capability.
Source: The latest AI scaling graph - and why it hardly makes sense
Assessment: MixedSaid
Observation
“METR’s blog and its tweets (perhaps written by publicists or by generative AI rather than by scientists), however, dropped the careful qualifications and nuance of the technical paper and made claims that go far beyond anything that the study actually supports.”
Our reading: METR's blog and tweets made claims far beyond what its study supports, dropping the technical paper's qualifications.
No horizon Evidence as of
Limitation: This assesses the identified announcements, not every METR tweet.
Source: The latest AI scaling graph - and why it hardly makes sense
Assessment: SupportedSaid
Observation
“Computers can answer some questions reliably now, for example, and some not, and the graph tells us nothing about which is which or when any specific question will be solved.”
Our reading: The METR graph tells us nothing about which questions computers can reliably answer or when specific questions will be solved.
No horizon Evidence as of
Limitation: The outcome depends on reading the claim as about specific questions.
Source: The latest AI scaling graph - and why it hardly makes sense
Assessment: UnresolvedSaid
Causal claim
“Claude does what LaMDA does better because it has more data, but I don’t see any really argument that Claude is any more sentient than a web browser.”
Our reading: Claude does what LaMDA does better because it has more data, but Marcus says he does not see any real argument that Claude is any more sentient than a web browser.
No horizon Evidence as of
Limitation: Private training information limits causal assessment.
Source: New adventures in AI hype: “Our language models are so ‘conscious’ we need to give them rights”
Assessment: SupportedSaid
Observation
“Anthropic is a business (which incidentally neglects to respect the rights of artist and writers who work they nick).”
Our reading: Anthropic fails to respect the rights of the artists and writers whose work it takes.
No horizon Evidence as of
Limitation: Support concerns documented book acquisition, not every creative work or every use.
Source: New adventures in AI hype: “Our language models are so ‘conscious’ we need to give them rights”
Assessment: UnresolvedSaid
Observation
“What these systems do, no more and no less, is to put together sequences of words, but without any coherent understanding of the world behind them, like foreign language Scrabble players who use English words as point-scoring tools, without any clue about what that mean.”
Our reading: LLMs only string together sequences of words and have no coherent understanding of the world behind them.
No horizon Evidence as of
Limitation: The positive mechanistic evidence comes from the model developer and has acknowledged methodological limits.
Source: New adventures in AI hype: “Our language models are so ‘conscious’ we need to give them rights”
Assessment: SupportedSaid
Observation
“the crown jewel that they reported on the demo — the 75% on Francois Chollet’s ARC test (once called ARC-AGI) doesn’t readily replicate.”
Our reading: OpenAI's reported 75% o3 score on ARC-AGI does not readily replicate.
No horizon Evidence as of
Limitation: Support depends on the released-model reading supplied by the surrounding text.
Source: OpenAI’s dirty December o3 demo doesn’t readily replicate
Assessment: UnresolvedSaid
Observation
“There were problems with experimental design, misleading graphs that left out competing work, and more.”
Our reading: OpenAI's December 2024 o3 demo had experimental design problems and misleading graphs that omitted competing work.
No horizon Evidence as of
Limitation: This is an evidence gap, not a finding that the presentation was sound.
Source: OpenAI’s dirty December o3 demo doesn’t readily replicate
Assessment: SupportedSaid
Observation
“OpenAI had privileged access to data their competitors didn’t have, but didn’t acknowledge this.”
Our reading: OpenAI had privileged access to FrontierMath data that competitors lacked and did not acknowledge it.
No horizon Evidence as of
Limitation: Epoch could evaluate competitors' models without giving their developers the underlying dataset.
Source: OpenAI’s dirty December o3 demo doesn’t readily replicate
Assessment: SupportedSaid
Observation
“without even acknowledging, for example, the massive problem LLMs have with fabricating citations”
Our reading: LLMs have a massive problem with fabricating citations.
No horizon Evidence as of
Limitation: Massive is a qualitative judgment.
Assessment: UnresolvedSaid
Observation
“Mollick’s notion of “jagged” AI is exactly what I used to mean by the term “pointillisitic” and what I predicted about o3 in November. AI that works some of the time but not always, markedly better on some problems than others.”
Our reading: o3 works only some of the time and is markedly better on some problems than others, as Marcus predicted in November 2024.
No horizon Evidence as of
Limitation: The performance evidence is company-reported.
Assessment: UnresolvedSaid
Observation
“my perspective that scaling was no longer working as effectively as it once did”
Our reading: Scaling of AI models is no longer working as effectively as it once did.
No horizon Evidence as of
Limitation: The evidence does not isolate pretraining scale from changes in data, architecture, and post-training across representative capabilities.
Assessment: UnresolvedSaid
Forecast
“debugging is hard, and a big part of what coders do, and not about to be replaced.”
Our reading: Debugging is hard and a big part of what coders do, and it is not about to be replaced by AI.
No deadline stated Evidence as of
Limitation: Survey responses about trust and usefulness are indirect measures of replacement.
Source: ”Those claiming we’re mere months away from AI agents replacing most programmers” should think again
Assessment: SupportedSaid
Observation
“It certainly took zero responsibility for what appears to be an obvious infringement.”
Our reading: Sam Altman's response at TED took no responsibility for what appears to be an obvious copyright infringement by an OpenAI system.
No horizon Evidence as of
Limitation: The finding concerns the content of the reply, not the legality of the image.
Source: Sam Altman’s attitude problem
Assessment: UnresolvedSaid
Observation
“NVidia is down significantly more this year (23.4%) than the S&P (11.5%).”
Our reading: NVidia's stock is down 23.4% in 2025 so far, significantly more than the S&P's 11.5% decline.
No horizon Evidence as of
Limitation: Primary historical records sought were inaccessible or returned the wrong period.
Source: Poor ROI for GenAI
Assessment: SupportedSaid
Causal claim
“GenAI is pitched as (near) AGI, but reliability problems persist, limiting how well it can be used in practice.”
Our reading: Although GenAI is marketed as near-AGI, its reliability problems persist and limit how well it can be used in practice.
No horizon Evidence as of
Limitation: The CEO's statement does not establish that existing products were AGI or that every supplier used the same framing.
Source: Poor ROI for GenAI
Assessment: SupportedSaid
Observation
“My sources assure me that this is NOT a clerical error and that Papan’s office confirms that the change is real.”
Our reading: The change replacing AB-501's content with unrelated provisions is real and not a clerical error, according to the author's sources.
No horizon Evidence as of
Limitation: The evidence does not identify Marcus's private sources or verify his communications with them.
Assessment: UnresolvedSaid
Observation
“Real world customers, many of whom are disappointed, are rarely mentioned.”
Our reading: Real-world customers, many of whom are disappointed, are rarely mentioned in the pundits' discussion of AI progress.
No horizon Evidence as of
Limitation: The survey was commissioned by an AI vendor and measures adoption disappointment, which can include organizational problems as well as product shortcomings.
Source: Deep Learning, Deep Scandal
Assessment: SupportedSaid
Observation
“First, the AI community is in fact pretty disappointed with LLama 4’s performance:”
Our reading: The AI community is disappointed with Llama 4's performance.
No horizon Evidence as of
Limitation: Public commentary is self-selected.
Source: Deep Learning, Deep Scandal
Assessment: ContradictedSaid
Observation
“If the juicing of benchmarks and “blending” of test data is the primary scandal, there is a secondary scandal too: a bunch of well-paid pundits, such as Roose, Newton, and Cowen, are failing to reckon with the repeated failures of massive companies to build GPT-5.”
Our reading: A number of well-paid pundits, such as Roose, Newton and Cowen, are failing to reckon with the repeated failures of massive companies to build GPT-5, which Marcus calls a secondary scandal next to benchmark juicing.
No horizon Evidence as of
Limitation: The result depends on reading reckoning as substantive consideration, rather than agreement about how much weight setbacks deserve.
Source: Deep Learning, Deep Scandal
Assessment: UnresolvedSaid
Observation
“Deep learning is indeed finally hitting a wall, in the sense of reaching a point of diminishing results.”
Our reading: Deep learning is hitting a wall in the sense of reaching diminishing returns.
No horizon Evidence as of
Limitation: METR evaluated an earlier GPT-4.5 checkpoint in a short, preliminary evaluation.
Source: Deep Learning, Deep Scandal
Assessment: SupportedSaid
Observation
“NVidia is down by a third, so far in 2025. (Far more than the stock market itself.)”
Our reading: Nvidia's stock fell by about a third in 2025 through early April, far more than the overall market.
No horizon Evidence as of
Limitation: The supported result depends on accepting approximate language: 30% is not exactly one third.
Source: Scaling is over, the bubble may be deflating, LLMs still can’t reason, and you can’t trust Sam
Assessment: SupportedSaid
Observation
“there still ain’t no GPT-5”
Our reading: As of April 2025, GPT-5 has not been released.
No horizon Evidence as of
Limitation: The assessment concerns public release, not the existence of internal models or private testing.
Source: Scaling is over, the bubble may be deflating, LLMs still can’t reason, and you can’t trust Sam
Assessment: SupportedSaid
Observation
“models still don’t reason reliably (as anticipated in What to Expect When Your Expecting GPT-4)”
Our reading: LLMs still do not reason reliably as of April 2025.
No horizon Evidence as of
Limitation: The cited versions are preprints.
Source: Scaling is over, the bubble may be deflating, LLMs still can’t reason, and you can’t trust Sam
Assessment: ContradictedSaid
Observation
“Scaling has run out (like I said it would)”
Our reading: Scaling of LLMs has run out as of April 2025.
No horizon Evidence as of
Limitation: The assessment interprets 'run out' as cessation of meaningful capability gains, rather than poor economic returns.
Source: Scaling is over, the bubble may be deflating, LLMs still can’t reason, and you can’t trust Sam
Assessment: SupportedSaid
Observation
“A new study on math, supporting what Davis and I wrote yesterday re LLMs struggling with mathematical reasoning from Mahdavi et al, converges on similar conclusions”
Our reading: A new study by Mahdavi et al. supports and reaches similar conclusions to what Marcus and Davis wrote about LLMs struggling with mathematical reasoning.
No horizon Evidence as of
Limitation: This comparison does not establish every broader assertion in the essay.
Source: Scaling is over, the bubble may be deflating, LLMs still can’t reason, and you can’t trust Sam
Assessment: UnresolvedSaid
Observation
“So Nestler’s experiment does not contradict the finding of the report; it corroborates it. o3, yet again, produced an invalid answer to this problem.”
Our reading: The o3 proof of USAMO 2025 problem 2 posted by Nestler was invalid, corroborating the report.
No horizon Evidence as of
Limitation: Search and page-access limitations may hide a public original or review.
Source: Reports of LLMs mastering math have been greatly exaggerated
Assessment: SupportedSaid
Observation
“The important problem is that they sometimes claim to they have solved problems that they have not properly solved; as with hallucinations, this reflects a deeply problematic failure to sanity check their own work.”
Our reading: LLMs sometimes falsely claim to have solved problems, reflecting a failure to sanity-check their own work.
No horizon Evidence as of
Limitation: The evidence concerns difficult Olympiad proofs and the tested model versions.
Source: Reports of LLMs mastering math have been greatly exaggerated
Assessment: SupportedSaid
Observation
“The results were dismal: None of the AIs scored higher than 5% overall.”
Our reading: On the 2025 USAMO, none of the leading reasoning LLMs tested scored higher than 5% overall.
No horizon Evidence as of
Limitation: This describes the original evaluation with its particular prompts, sampling procedure and expert grading.
Source: Reports of LLMs mastering math have been greatly exaggerated
Assessment: SupportedSaid
Causal claim
“But that is because they rely in part on powerful, completely hand-written, symbolic reasoning systems. LLMs are not similarly immune.”
Our reading: AlphaProof and AlphaGeometry cannot generate incorrect proofs because they rely in part on powerful, completely hand-written symbolic reasoning systems, and LLMs are not similarly immune.
No horizon Evidence as of
Limitation: The result depends on distinguishing accepted proofs from generated candidates.
Source: Reports of LLMs mastering math have been greatly exaggerated
Assessment: UnresolvedSaid
Observation
“Importantly, the neurosymbolic method used by DeepMind’s AlphaProof and AlphaGeometry systems (which we discussed recently) which (more or less) achieved a silver-medal level performance on the 2024 International Math Olympiad, is immune to this problem.”
Our reading: DeepMind's neurosymbolic AlphaProof and AlphaGeometry are immune to producing confidently incorrect proofs.
No horizon Evidence as of
Limitation: The result depends materially on distinguishing AlphaGeometry from AlphaGeometry 2.
Source: Reports of LLMs mastering math have been greatly exaggerated
Assessment: UnresolvedSaid
Observation
“An LLM just spits back (sometimes in transmogrified form) what it has read.”
Our reading: An LLM just spits back what it has read, sometimes in transmogrified form.
No horizon Evidence as of
Limitation: Unresolved does not mean that the evidence favors the claim.
Assessment: UnresolvedSaid
Observation
“Humans are (usually, if well-motivated) smart enough to sort things like jokes and fiction from reality, but current AI doesn’t have a clue.”
Our reading: Current AI systems cannot tell jokes and fiction apart from reality, while well-motivated humans usually can.
No horizon Evidence as of
Limitation: The cited studies do not establish either representative failures or a usable ability on the specified fiction-versus-fact task during the assessment period.
Assessment: SupportedSaid
Observation
“In Taming Silicon Valley, I warned that LLMs would be used for dumb things that would affect lots of people.”
Our reading: Marcus says that in Taming Silicon Valley he warned that LLMs would be used for dumb things that would affect lots of people, and he presents the tariff formula, which he says was probably recommended by an LLM, as confirming that warning.
No horizon Evidence as of
Limitation: The complete book was not inspected; the assessment rests on an identified excerpt and an independent review.
Assessment: UnresolvedSaid
Observation
“I do genuinely believe today’s Turing Test takers are better AI systems than an earlier generation. I hardly think that means anything is “over”.”
Our reading: Today's Turing Test-taking AI systems are better than the earlier generation, but that does not mean anything is 'over'.
No horizon Evidence as of
Limitation: This is an evidence gap, not evidence that contemporary systems lack broader improvements.
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: UnresolvedSaid
Conditional forecast
“But I am willing to wager that if someone put in that kind of effort and carefully built out the test, no current system would come close to human performance.”
Our reading: If a careful comprehension challenge test were built, no current AI system would come close to human performance on it.
No deadline stated Evidence as of
Only if: if someone carefully built out the comprehension challenge test
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: SupportedSaid
Causal claim
“To do well the system had to be coached to adopt the persona of “a young person who is introverted, knowledgeable about internet culture, and uses slang”; it’s still in no small part about the tricks.”
Our reading: To do well in the new study, the system had to be coached to adopt the persona of a young, introverted, slang-using person knowledgeable about internet culture, so its success is still in no small part about tricks.
No horizon Evidence as of
Limitation: The comparison does not establish that this particular persona was necessary among all possible prompts [e1].
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: UnresolvedSaid
Observation
“Unfortunately, nobody has ever fully implemented the comprehension challenge test that I proposed.”
Our reading: Nobody has fully implemented Marcus's proposed comprehension challenge test.
No horizon Evidence as of
Limitation: The boundary between a related benchmark and full implementation is not formally defined.
Source: AI has (sort of) passed the Turing Test; here’s why that hardly matters
Assessment: SupportedSaid
Observation
“But in an initial public offering yesterday CoreWeave valued themselves at just $23 billion - a major retreat.”
Our reading: CoreWeave's IPO valued it at about $23 billion, a major retreat from the roughly $32 billion it was thought to be worth weeks earlier.
No horizon Evidence as of
Limitation: The fully diluted valuations rely on reputable news reporting; the company release confirms pricing but does not state those total valuations.
Source: Unwoven
Assessment: UnresolvedSaid
Forecast
“Agents will be a disappointment, precisely because of all of the ongoing reliability and hallucination issues.”
Our reading: AI agents will be a disappointment because of ongoing reliability and hallucination problems.
No deadline stated Evidence as of
Limitation: The evidence mixes enterprise decision-makers, agent developers and deployed-system practitioners.
Assessment: SupportedSaid
Observation
“Especially since DeepSeek there has been a massive price war.”
Our reading: There has been a massive LLM price war, especially since DeepSeek.
No horizon Evidence as of
Limitation: Historical primary announcements were not recovered for the decisive reports; reputable contemporaneous Reuters reporting is used.
Assessment: UnresolvedSaid
Forecast
“For most GenAI model building companies, profits will be meager.”
Our reading: Most GenAI model-building companies will have meager or no profits.
No deadline stated Evidence as of
Limitation: The evidence concerns one company's projected losses before the claim was published, not comparable realized results for a defined population [e1].
Assessment: UnresolvedSaid
Observation
“This And the promised major jumps in economic productivity simply haven’t emerged.”
Our reading: The promised major gains in economic productivity from GenAI have not materialized.
No horizon Evidence as of
Limitation: The US-focused evidence does not establish a worldwide result or isolate GenAI's aggregate contribution in the target quarter [e1, e2].
Assessment: UnresolvedSaid
Observation
“(Or nil; most are losing money; few have made any profits at all).”
Our reading: Most GenAI model-building companies are losing money and few have made any profit.
No horizon Evidence as of
Limitation: Anthropic's historical results were disclosed after the post but before the evaluation date.
Assessment: UnresolvedSaid
Forecast
“The end game here is not that LLMs will vanish, but that LLMs will be virtually free.”
Our reading: LLMs will not vanish but will ultimately become virtually free.
No deadline stated Evidence as of
Limitation: Consumer chat, API inference and downloadable weights have different access costs.
Assessment: SupportedSaid
Observation
“GenAI is an approach that is enormously popular, but one that is neither reliable nor particularly well-grounded in truth.”
Our reading: GenAI is an enormously popular approach, but one that is neither reliable nor particularly well-grounded in truth.
No horizon Evidence as of
Limitation: The adoption figure is company-reported through a news interview, not independently audited.
Assessment: UnresolvedSaid
Observation
“User disillusionment continues to grow.”
Our reading: User disillusionment with generative AI is continuing to grow.
No horizon Evidence as of
Limitation: The satisfaction comparison covers shopping-related use in 2024, not the target period in early 2025 [e1].
Assessment: UnresolvedSaid
Observation
“NVidia, the largest company actually making serious profits from AI, is down 17.75% for the year, and (after a meteoric rise) clearly facing headwinds.”
Our reading: Nvidia, the largest company making serious AI profits, is down 17.75% year-to-date in 2025 and clearly facing headwinds.
No horizon Evidence as of
Limitation: Several promising market-data records could not be read with the required tool.
Assessment: UnresolvedSaid
Forecast
“GenAI is never going to disappear.”
Our reading: Generative AI will never disappear.
No deadline stated Evidence as of
Limitation: The survey covers US Gen Z respondents, not all populations or deployments [e1].
Assessment: UnresolvedSaid
Observation
“Thanks to a search tool provided by the Atlantic, I discovered tonight that in this one database alone Meta pirated over 100 of my books and scientific articles.”
Our reading: Meta pirated over 100 of Gary Marcus's books and scientific articles via the Libgen database alone.
No horizon Evidence as of
Limitation: An inaccessible source is an evidence gap, not evidence that Marcus was wrong.
Source: Meta pirated at least 101 of my books and articles, and tens of millions of others
Assessment: SupportedSaid
Observation
“Many other authors are discovering the same thing; social media is full of reports like mine.”
Our reading: Many other authors are discovering that Meta used their works from the pirated database.
No horizon Evidence as of
Limitation: The evidence establishes widespread reports qualitatively, not a measured share of social media.
Source: Meta pirated at least 101 of my books and articles, and tens of millions of others
Assessment: SupportedSaid
Observation
“He worked hard on every one of those books and articles and does not appreciate them being used without permission by AI systems with an established tendency towards regurgitation.”
Our reading: AI systems such as Meta's have an established tendency to regurgitate their training data.
No horizon Evidence as of
Limitation: The studies do not establish how often reproduction occurs in representative everyday conversations or whether Marcus's particular works were reproduced [e1, e2].
Source: Meta pirated at least 101 of my books and articles, and tens of millions of others
Assessment: UnresolvedSaid
Observation
“In every single case, they used them without my permission.”
Our reading: Meta used all of Marcus's works in the database without his permission.
No horizon Evidence as of
Limitation: The cited guidance addresses books generally and does not inventory Marcus's scientific articles.
Source: Meta pirated at least 101 of my books and articles, and tens of millions of others
Assessment: SupportedSaid
Observation
“I don’t want spoil all the fun, but … things didn’t go well:”
Our reading: In The Guardian's test, people using AI tools without coding knowledge did not succeed well at building a Pac-Man clone.
No horizon Evidence as of
Limitation: The report does not establish universal failure or a population failure rate [e1, e2].
Source: AI Coding Fantasy meets Pac-Man
Assessment: SupportedSaid
Forecast
“To be sure, most or all trained coders will (if they are not already) soon use AI as a part of their workflow.”
Our reading: Most or all trained coders will soon use AI as part of their workflow, if they are not already.
Settled before its deadline Evidence as of
Limitation: These are self-reported surveys, not a census.
Source: AI Coding Fantasy meets Pac-Man
Assessment: UnresolvedSaid
Forecast
“AI will be a tool to help people write code, just as spell-checkers are a tool to help authors write articles and novel, but AI will not soon replace people who understand how to conceive of, write, and debug code.”
Our reading: AI will serve as a tool to help people write code but will not soon replace people who know how to conceive of, write, and debug code.
No deadline stated Evidence as of
Limitation: The available employment research does not directly measure responsibility for conceiving and debugging software.
Source: AI Coding Fantasy meets Pac-Man