Supported claims by Gary Marcus, page 4 of 7
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Supported claims by Gary Marcus (694)
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Page 4 of 7: claims 301 to 400 of 694, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Forecast
“No doubt the next generation of systems will have greater abilities and more extensive applications and will somewhat mitigate the weaknesses.”
Our reading: The next generation of AI systems will have greater abilities and broader applications and will somewhat mitigate current weaknesses.
Settled before its deadline Evidence as of
Limitation: Support uses the ordinary directional reading, not a claim that every successor or application improved.
Source: The Five Stages of AGI Grief
Assessment: SupportedSaid
Observation
“While such a milestone would be important economically and socially, it is also a departure from, and even a retrenchment from, the kind of cognitive criteria around flexibility and resourcefulness that people in the field had generally talked about it.”
Our reading: A milestone of a system doing 80% of economically valuable human work, which Marcus recalls (not with certainty) OpenAI suggesting as a definition of AGI, would be important economically and socially, but it departs from, and even retreats from, the cognitive criteria of flexibility and resourcefulness that people in the field had generally discussed.
No horizon Evidence as of
Limitation: Support applies to the comparison of stated requirements.
Source: The Five Stages of AGI Grief
Assessment: SupportedSaid
Observation
“The problem of distribution shift, well-tested in Apple’s 2024 reasoning paper, which goes back to my own 1998 work, continues.”
Our reading: The problem of distribution shift in neural models continues to be unsolved.
No horizon Evidence as of
Limitation: Support concerns continued empirical failures, not a proof that neural models cannot generalize.
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: SupportedSaid
Observation
“Commonsense reasoning is still dodgy, as it has long been, as Ernie Davis and I just reviewed here..”
Our reading: Commonsense reasoning in current AI systems remains unreliable, as it long has been.
No horizon Evidence as of
Limitation: Commonsense is broader than the tasks tested.
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: SupportedSaid
Observation
“LLMs, like their predecessors generalize well to similar items but struggle with reliability in unfamiliar regimes, even minor variations sometimes break them.”
Our reading: LLMs generalize well to similar items but are unreliable in unfamiliar regimes, and minor variations sometimes break them.
No horizon Evidence as of
Limitation: The cited evidence concerns mathematical word problems, not every application of LLMs.
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: SupportedSaid
Observation
“Many leading figures in the field have acknowledged that we may have reached a period of diminishing returns of pure LLM scaling, much as I anticipated in 2022.”
Our reading: Many leading figures in the field have acknowledged that pure LLM scaling may have reached a period of diminishing returns, much as Marcus anticipated in 2022.
No horizon Evidence as of
Limitation: The quantity 'many' remains interpretive.
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: SupportedSaid
Forecast
“They will be a major focus of attention in the field in 2025.”
Our reading: AI agents will be a major focus of attention in the AI field in 2025.
Deadline passed Evidence as of
Limitation: 'Major focus' has no numeric threshold. The evidence shows agents were prominent but does not rank them against other themes in the field.
Assessment: SupportedSaid
Observation
“But what was most striking to us was not the unsurprising fact that these systems are not close to ready for prime time (an employee that bad would probably be immediately fired), but the examples from the list of what went wrong:”
Our reading: Current LLM agents (Claude, Gemini, GPT-4 etc.) are not close to ready for prime time on real-world workplace tasks.
No horizon Evidence as of
Limitation: 'Ready for prime time' has no explicit threshold and is a judgment.
Assessment: SupportedSaid
Observation
“Ten months later, Sora is finally available to the public, and it’s apparent that it too is struggling with physical reasoning.”
Our reading: Sora, now publicly available, struggles with physical reasoning.
No horizon Evidence as of
Limitation: The announcement does not quantify the physical-error rate or independently reproduce the examples in the post.
Assessment: SupportedSaid
Observation
“But one important component of physical and conceptual understanding is reasoning about entities and their properties, and current approaches have consistently fallen short on this.”
Our reading: Current AI approaches have consistently fallen short on reasoning about entities and their properties.
No horizon Evidence as of
Limitation: This outcome depends on reading consistently as recurring gaps, not failure on every task.
Assessment: SupportedSaid
Observation
“Depressingly, despite all of the apparent progress in the last few years, much of what we raised in 2015 and re-emphasized in 2019 remains unsolved.”
Our reading: Much of the commonsense reasoning problem the authors raised in 2015 and 2019 remains unsolved in AI as of January 2025.
No horizon Evidence as of
Limitation: The result depends on interpreting 'unsolved' as lacking broad reliability.
Assessment: SupportedSaid
Observation
“If general purpose domestic robots are a long way away, “AI agents” —AI systems that act in the world on user’s behalf—are already here to a limited degree.”
Our reading: AI agents, meaning AI systems that act in the world on users' behalf, are already here to a limited degree, even though general-purpose domestic robots are a long way away.
No horizon Evidence as of
Limitation: This establishes limited availability and functionality, not general reliability or widespread adoption.
Assessment: SupportedSaid
Observation
“Hoping that reliable commonsense will simply “emerge” from large databases and video has proven unrealistic, again and again.”
Our reading: The hope that reliable commonsense will emerge from large text and video datasets has repeatedly proven unrealistic.
No horizon Evidence as of
Limitation: Reliable commonsense remains an interpretive standard.
Assessment: SupportedSaid
Forecast
“AI “Agents” will be endlessly hyped throughout 2025 but far from reliable, except possibly in very narrow use cases.”
Our reading: Throughout 2025, AI agents will be endlessly hyped but far from reliable, except possibly in very narrow use cases.
Deadline passed Evidence as of
Limitation: The promotion shown comes from one company, OpenAI. It does not measure how much agents were promoted across the industry.
Assessment: SupportedSaid
Forecast
“Copyright lawsuits over generative AI will continue throughout the year.”
Our reading: Copyright lawsuits over generative AI will continue throughout 2025.
Deadline passed Evidence as of
Limitation: The records establish continuing litigation, not the merits of infringement allegations.
Assessment: SupportedSaid
Forecast
“Reasoning flubs will continue to haunt generative AI.”
Our reading: Through 2025 reasoning errors will continue to plague generative AI.
Deadline passed Evidence as of
Limitation: 'Flubs' and 'haunt' set no threshold, so the verdict rests on a plain reading.
Assessment: SupportedSaid
Observation
“Immediately after the February announcement of Sora, I noted Sora’s troubles with physics and predicted that they wouldn’t go away. They haven’t.”
Our reading: Sora's troubles with physics have not gone away since its February 2024 announcement.
No horizon Evidence as of
Limitation: The disclosures do not quantify failure rates or establish that physics performance had not improved [e1, e2].
Assessment: SupportedSaid
Observation
“My prediction that 2024 would be seen as the year of AI disillusionment was overstated but not entirely off the mark, holding true for some folks but not all, and as such perhaps deserve partial credit.”
Our reading: 2024 was a year of AI disillusionment for some but not all; his prediction was overstated.
No horizon Evidence as of
Limitation: The surveys cover business leaders and employees rather than a representative global public.
Assessment: SupportedSaid
Observation
“There is still no robust solution to hallucinations.”
Our reading: There is still no robust solution to LLM hallucinations.
No horizon Evidence as of
Limitation: The main source is a lab paper describing its own models' limits. Independent leaderboard figures for late 2024 could not be read.
Assessment: SupportedSaid
Forecast
“Hallucinations (which should really be called confabulations) will continue to haunt generative AI.”
Our reading: Through 2025 hallucinations will continue to plague generative AI.
Deadline passed Evidence as of
Limitation: 'Haunt' has no threshold. The verdict rests on the plain reading that the problem stayed material and recognized.
Assessment: SupportedSaid
Observation
“There are many such models now; there is a price war. There is very little moat.”
Our reading: There are many GPT-4 level models, a price war, and very little moat among LLM providers.
No horizon Evidence as of
Limitation: 'Moat' is read as a lasting performance lead. Brand, distribution or user base, which are other possible moats, were not measured.
Assessment: SupportedSaid
Forecast
“AI “Agents” will be endlessly hyped throughout 2025 but far from reliable, except possibly in very narrow use cases.”
Our reading: AI agents will be heavily hyped in 2025 but far from reliable except possibly in very narrow use cases.
Deadline passed Evidence as of
Limitation: The result partly depends on reading coding agents as a 'narrow use case'. A reader who treats coding as broad could call this mixed.
Assessment: SupportedSaid
Forecast
“OpenAI will continue to preview products months and perhaps years before they are solid and widely available at an accessible price.”
Our reading: By the end of 2025 OpenAI will have continued to preview products months, and perhaps years, before they are solid and widely available at an accessible price.
Deadline passed Evidence as of
Limitation: The result depends on treating premium Pro-only previews as short of ordinary broad consumer access.
Assessment: SupportedSaid
Observation
“At *most* the test is necessary for AGI; it certainly isn’t sufficient.”
Our reading: Passing ARC is at most necessary for AGI and certainly not sufficient.
No horizon Evidence as of
Limitation: The contract notes that AGI is not defined, so 'sufficient' is judged by the designers' stated view.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: SupportedSaid
Observation
“What was actually done - pretraining on what I believe was hundreds of public examples - is NOT comparable to what humans require.”
Our reading: What was done for o3 on ARC, pretraining on what Marcus believes was hundreds of public examples, is not comparable to what humans require.
No horizon Evidence as of
Limitation: The conclusion depends on reading comparable as referring to ARC-specific preparation.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: SupportedSaid
Observation
“We still don’t have a solid test of what o3 does without the pretraining, in the case that would be more comparable to humans”
Our reading: There is no solid test yet of o3's ARC performance without pretraining.
No horizon Evidence as of
Limitation: The evaluator page has later updates, and the training note's precise insertion date is not separately archived in this record.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: SupportedSaid
Observation
“I am saying that many basic problems haven’t been solved.”
Our reading: Many basic problems on the path to AGI remain unsolved.
No horizon Evidence as of
Limitation: The scope and meaning of 'many' are inferred and materially affect the assessment.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Observation
“The product was not yet released to the general public; it’s not been vetted by the scientific community.”
Our reading: As of the demo, o3 had not been released to the public nor vetted by the scientific community.
No horizon Evidence as of
Limitation: 'Vetted by the scientific community' has no precise test. ARC Prize's evaluation is a partial independent check, which he acknowledged.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Observation
“The best results were truly impressive, such as major advances on Francois Chollet’s ARC task and on an important mathematical benchmark FrontierMath.”
Our reading: o3's best demo results showed truly impressive major advances on ARC and FrontierMath.
No horizon Evidence as of
Limitation: 'Truly impressive' is a judgment and was not scored.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Observation
“(It certainly was not the long-awaited GPT-5).”
Our reading: OpenAI's o3 is not the long-awaited GPT-5.
No horizon Evidence as of
Limitation: The core evidence for the demo day is a news report, not an OpenAI page from that day.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Observation
“As in: Passing ARC may be necessary but certainly not sufficient for AGI.”
Our reading: Passing the ARC test is certainly not sufficient for AGI, though it may be necessary.
No horizon Evidence as of
Limitation: The verdict concerns what the benchmark result establishes, not a universal theorem about every possible AGI definition.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: SupportedSaid
Observation
“OpenAI’s problems with text and copyright started to become especially clear a year ago, when the New York Times sued them, and showed how close some of the outputs could be to some of the stories on their website.”
Our reading: OpenAI's problems with text and copyright started to become especially clear a year ago, when the New York Times sued OpenAI and showed how close some of the outputs could be to some of the stories on its website.
No horizon Evidence as of
Limitation: A complaint establishes allegations and presented evidence, not a judgment of infringement.
Source: Generative AI’s Continuing Copyright Problems, an Essay in Memory of Suchir Balaji, 1998 - 2024
Assessment: SupportedSaid
Observation
“They can easily tell you one thing and (especially if hooked up as agents) do another — possibly quite contrary to what they have alleged they are doing.”
Our reading: LLMs can easily say one thing and do another, especially when deployed as agents.
No horizon Evidence as of
Limitation: Follow-up results concern trials where the relevant actions had already occurred; they do not measure how often those actions occur in ordinary use [e1].
Source: Humanity’s “Oh shit!” AI moment?
Assessment: SupportedSaid
Observation
“In 2022 I warned that there would be chatbot related deaths such as suicides. This began to happen in 2023 and continued in 2024.”
Our reading: Chatbot-related deaths such as suicides occurred in 2023 and 2024.
No horizon Evidence as of
Limitation: Causation is not established. The Belgian link rests on the widow's account and chat logs she supplied. The Setzer link rests on lawsuit allegations.
Source: Casey (Newton) Strikes Out
Assessment: SupportedSaid
Observation
“Part of the value in skepticism is correctly anticipating road blocks, and suggesting alternative directions (as I have for decades with neurosymbolic AI, which is at last coming into vogue, as noted in Fortune earlier this week).”
Our reading: Neurosymbolic AI is finally coming into vogue.
No horizon Evidence as of
Limitation: The result depends on reading vogue as growing attention.
Source: Casey (Newton) Strikes Out
Assessment: SupportedSaid
Observation
“That gross violation of spatiotemporal continuity/failure of object permanence is not a one-off, either; it’s something general.”
Our reading: Sora's failures of object permanence and spatiotemporal continuity are general rather than isolated.
No horizon Evidence as of
Limitation: Support rests on contemporaneous developer documentation rather than a representative independent audit.
Assessment: SupportedSaid
Observation
“Two years later Google’s search share hasn’t diminished all that much.”
Our reading: Two years after ChatGPT's launch, Google's search share has not diminished much.
No horizon Evidence as of
Limitation: The figures come from secondary reporting of Statcounter because historical primary data were not retrievable.
Source: ChatGPT, at Age Two
Assessment: SupportedSaid
Observation
“GPT-4 was better than GPT-3 in almost every way; o1 is a far less uniform improvement.”
Our reading: GPT-4 was better than GPT-3 in almost every way, whereas o1 is a far less uniform improvement.
No horizon Evidence as of
Limitation: The result depends on reading the comparison qualitatively across capability categories.
Assessment: SupportedSaid
Observation
“(In some cases o1 doesn’t even do better than GPT-4).”
Our reading: In some cases o1 does not perform better than GPT-4.
No horizon Evidence as of
Limitation: The supported comparison is principally against GPT-4o, not specifically the original GPT-4.
Assessment: SupportedSaid
Observation
“Many other Tech CEOs and others, including Marc Andreeseen (A16Z), Alexandr Wang (ScaleAI), have also used the phrase “hitting a wall” and the idea of diminishing returns in the last couple weeks.”
Our reading: In the last couple of weeks before November 23, 2024, many other tech CEOs and others, including Marc Andreessen and Alexandr Wang, used the phrase "hitting a wall" and the idea of diminishing returns.
No horizon Evidence as of
Limitation: Andreessen did not use the exact phrase 'hitting a wall'; the claim is scored on the group reading.
Source: Satya Nadella and the three stages of scientific truth
Assessment: SupportedSaid
Observation
“Scaling was always a hypothesis.”
Our reading: Scaling (improving AI by adding more data and compute) was always a hypothesis.
No horizon Evidence as of
Limitation: Hypothesis does not mean unsupported speculation.
Assessment: SupportedSaid
Observation
“They are merely empirical generalizations that held for a certain period time, when there was enough fresh data and compute.”
Our reading: Scaling laws are merely empirical generalizations that held for a limited period while enough fresh data and compute were available.
No horizon Evidence as of
Limitation: The result depends on reading the sentence in light of the surrounding hedges about scaling having stopped.
Assessment: SupportedSaid
Observation
“Many of us, including Melanie Mitchell, Subbarao Kambahapati, Emily Bender, Ernie Davis, etc. have been emphasizing for ages that there are principled limits with LLMs.”
Our reading: Marcus and many others, including Melanie Mitchell, Subbarao Kambhampati, Emily Bender and Ernie Davis, have long emphasized that there are principled limits with LLMs.
No horizon Evidence as of
Limitation: 'For ages' is colloquial, and the evidence does not establish an identical duration for every researcher.
Source: CONFIRMED: LLMs have indeed reached a point of diminishing returns
Assessment: SupportedSaid
Observation
“People aren’t ignoring GenAI; they are waiting to see if it will work.”
Our reading: People are not ignoring generative AI; they are waiting to see whether it will work.
No horizon Evidence as of
Limitation: 'People' is undefined; the strongest evidence covers businesses.
Source: AI influencer hype gone wild
Assessment: SupportedSaid
Observation
“(is OpenAI’s o1 100 times better than GPT4? Certainly not by their own data)”
Our reading: By OpenAI's own data, o1 is certainly not 100 times better than GPT-4.
No horizon Evidence as of
Limitation: OpenAI's baseline is GPT-4o, not the original GPT-4.
Assessment: SupportedSaid
Observation
“Jailbreaks aren’t new, but even after years of them, the tech industry has nothing like a robust response.”
Our reading: Despite years of jailbreaks, the tech industry has no robust defense against them.
No horizon Evidence as of
Limitation: The result depends on the stated reading of robust.
Source: When it comes to security, LLMs are like Swiss cheese — and that’s going to cause huge problems
Assessment: SupportedSaid
Observation
“Fall off on increasingly large multiplication problems has repeatedly been observed, both in older models and newer models.”
Our reading: Both older and newer LLMs have repeatedly been observed to get worse as multiplication problems get larger.
No horizon Evidence as of
Limitation: The Deng results are a researcher's social-media post plus a news report, not a peer-reviewed paper.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Observation
“Performance is ok on small problems, but quickly tails off.”
Our reading: o1 does acceptably on small planning problems, but its performance drops quickly as problems get larger.
No horizon Evidence as of
Limitation: The model tested was o1-preview.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Observation
“Failure to follow the rules of chess is another continuing failure of formal reasoning:”
Our reading: LLMs still fail to follow the rules of chess, which is an ongoing failure of formal reasoning.
No horizon Evidence as of
Limitation: The evidence is one puzzle benchmark with a single prompt format. Other formats or full games may give different illegal-move rates.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Observation
“Two recent studies show nothing like “10x” improvement :”
Our reading: Two recent studies of generative AI coding tools show nothing like a 10x productivity improvement.
No horizon Evidence as of
Limitation: These findings concern the two identified studies and do not establish effects for all coding tools, developers or later versions.
Source: Sorry, GenAI is NOT going to 10x computer programming
Assessment: SupportedSaid
Observation
“The data are coming in – and it’s not.”
Our reading: Emerging data show generative AI is not improving programmer productivity by a factor of 10.
No horizon Evidence as of
Limitation: The studies use different productivity measures and settings.
Source: Sorry, GenAI is NOT going to 10x computer programming
Assessment: SupportedSaid
Observation
“many lawsuits pending”
Our reading: OpenAI faces many pending lawsuits.
No horizon Evidence as of
Limitation: Five is an evaluator-supplied threshold, not a number Marcus stated.
Source: OpenAI’s slow-motion train wreck
Assessment: SupportedSaid
Observation
“From left to right that’s Ilya Sutskever (now gone, less than a year later), Greg Brockman (on leave, at least until the end of the year), CTO Mira Murati (departure just announced) and Sam Altman (fired, and then rehired).”
Our reading: Of the four OpenAI leaders on the cover, Ilya Sutskever is gone less than a year later, Greg Brockman is on leave at least until the end of the year, CTO Mira Murati's departure was just announced, and Sam Altman was fired and then rehired.
No horizon Evidence as of
Limitation: The eventual duration of Brockman's leave differed from the announced plan.
Source: OpenAI’s slow-motion train wreck
Assessment: SupportedSaid
Observation
“Co-founders John Schulman also left, so did Andrej Karpathy and Jan Leike, and perhaps a dozen others.”
Our reading: John Schulman, Andrej Karpathy, Jan Leike and perhaps a dozen others have left OpenAI.
No horizon Evidence as of
Limitation: The additional count remains unverified and lacks a defined counting period.
Source: OpenAI’s slow-motion train wreck
Assessment: SupportedSaid
Observation
“I rushed out a hot take on X, my usual cautions about how LLMs aren’t really AGI:”
Our reading: LLMs, including OpenAI's new o1, are not really AGI.
No horizon Evidence as of
Limitation: The system card tests several model checkpoints and configurations; it is not an exhaustive evaluation of all contemporary LLMs [e3].
Assessment: SupportedSaid
Observation
“Almost exactly what I predicted we would see with GPT-4, back on Christmas Day 2022.”
Our reading: Altman's description of o1 as 'still flawed, still limited, seem more impressive on first use' is almost exactly what Marcus predicted we would see with GPT-4 on Christmas Day 2022.
No horizon Evidence as of
Limitation: The agreement is thematic, not verbatim.
Assessment: SupportedSaid
Causal claim
“You should be watching to see what the candidates plan to do about fraud and disinformation, as AI contributes to escalation in both.”
Our reading: AI is contributing to an escalation in fraud and disinformation.
No horizon Evidence as of
Limitation: The evidence does not quantify AI-attributable growth in aggregate fraud losses or successful deception [e1, e2].
Source: What will the candidates say about AI in tonight’s debate?
Assessment: SupportedSaid
Observation
“And you should be watching to see what they plan to do about the growing problem of nonconsensual deep fake porn, and especially how that might affect the already vulnerable population of teenage girls, who have been hard by social media.”
Our reading: Nonconsensual deepfake porn is a growing problem.
No horizon Evidence as of
Limitation: The Korean police figures do not establish worldwide prevalence or separate increased offending from increased recognition and reporting [e1].
Source: What will the candidates say about AI in tonight’s debate?
Assessment: SupportedSaid
Observation
“California’s pending AI bill SB-1047 was supposed to be a tiny step in that direction, and even that has gotten watered down.”
Our reading: California's SB-1047, already only a small step toward external AI governance, has been watered down.
No horizon Evidence as of
Limitation: The evidence is the author's dated amendment explanation, not a complete comparison of all bill versions.
Source: Scoop: What former employees of OpenAI are worried about
Assessment: SupportedSaid
Observation
“When Sam Altman stood next to me in the Senate, he told the Senate that he strongly favored AI regulation. But behind the scenes his team fought to water down the EU AI Act.”
Our reading: Sam Altman told the Senate, standing next to Marcus, that he strongly favored AI regulation, but behind the scenes his team fought to water down the EU AI Act.
No horizon Evidence as of
Limitation: The lobbying finding relies on reputable reporting because the available primary PDF extraction is corrupted.
Assessment: SupportedSaid
Observation
“It also provides important whistleblower protections, which as many ex OpenAI employees have made clear, are critical.”
Our reading: SB-1047 provides important whistleblower protections, which, as many ex-OpenAI employees have made clear, are critical.
No horizon Evidence as of
Limitation: The assessment concerns a proposed bill, not protections already in force.
Assessment: SupportedSaid
Observation
“Most or all of the major big tech companies joined a lobbying organization that fought SB-1047, tooth and nail, despite broad public support for the bill.”
Our reading: Most or all major big tech companies joined a lobbying group that fought SB-1047 hard despite broad public support for it.
No horizon Evidence as of
Limitation: The majority assessment uses the contract's five-company group: Alphabet/Google, Amazon, Apple, Meta, and Microsoft.
Assessment: SupportedSaid
Observation
“California’s SB-1047, not yet signed into law, but on its way to being one of the first really substantive AI bills in the US, primarily addressed to liability around catastrophic risks, was significantly weakened in last-minute negotiations.”
Our reading: California's SB-1047 was significantly weakened in last-minute negotiations.
No horizon Evidence as of
Limitation: Significance is an evaluative judgment about the documented legal changes.
Assessment: SupportedSaid
Causal claim
“As I emphasized last year in front of the Senate, there is not one risk but many: bias, disinformation, defamation, toxicity, invasion of privacy, and so much more, to say nothing of risks to employment, intellectual property, and the environment, precisely because recent techniques can be applied in so many ways, still poorly understood.”
Our reading: AI poses not one risk but many, including bias, disinformation, defamation, toxicity, invasion of privacy and so much more, plus risks to employment, intellectual property and the environment, precisely because recent techniques can be applied in so many ways and are still poorly understood.
No horizon Evidence as of
Limitation: The evidence includes scientific syntheses and a preprint, rather than one experiment testing the entire claim.
Assessment: SupportedSaid
Observation
“Lastly, asking for standards is not unique to AI; it's common across many industries to ask that companies evaluate the safety of their products according to set standards: just look at the pharmaceutical industry, or aviation, automobiles, etc.”
Our reading: Asking for standards is not unique to AI: in many industries, such as pharmaceuticals, aviation and automobiles, it is common to ask companies to evaluate the safety of their products against set standards.
No horizon Evidence as of
Limitation: The evidence establishes the three named U.S. examples, not a numerical survey of every industry or jurisdiction.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047
Assessment: SupportedSaid
Observation
“And much of what would be required is already something companies committed to voluntarily, in discussions at the White House and in Seoul.”
Our reading: Much of what SB-1047 would require is something AI companies already voluntarily committed to at the White House and in Seoul.
No horizon Evidence as of
Limitation: The assessment is a qualitative comparison, not a count of statutory clauses.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047
Assessment: SupportedSaid
Observation
“OpenAI recently put Paul Nakasone (ex NSA) on the board”
Our reading: OpenAI recently put Paul Nakasone, formerly of the NSA, on its board.
No horizon Evidence as of
Limitation: This establishes the appointment and career background, not any implication about surveillance.
Source: OpenAI: On a Path to Becoming The World’s Most Frightening Surveillance Company?
Assessment: SupportedSaid
Observation
“An awful lot of key players have left the island. Most voluntarily.”
Our reading: Many key OpenAI people have left, most of them voluntarily.
No horizon Evidence as of
Limitation: The roster is a transparent group drawn from contemporaneous reporting, not a complete personnel register.
Source: The OpenAI Plot Thickens
Assessment: SupportedSaid
Observation
“Nvidia dropped 6%, 20% over the last month”
Our reading: Nvidia stock fell 6% on August 5, 2024, and 20% over the previous month.
No horizon Evidence as of
Limitation: The result depends on reading the figures as rounded closing-price changes.
Assessment: SupportedSaid
Observation
“OpenAI tempered expectations for its next event, and said we wouldn’t see GPT-5 then.”
Our reading: OpenAI tempered expectations for its next event and said we would not see GPT-5 at that event.
No horizon Evidence as of
Limitation: Evidence is a reporter's direct quotation, not an accessible original company announcement.
Assessment: SupportedSaid
Observation
“Over a quarter century later, handling outliers is still the Achilles’ Heel of neural networks.”
Our reading: More than a quarter century after his 1998 work, handling outliers (distribution shift) is still the Achilles' heel of neural networks.
No horizon Evidence as of
Limitation: The decisive study concerns multimodal language models rather than every neural architecture.
Source: This one important fact about current AI explains almost everything
Assessment: SupportedSaid
Observation
“Another GenAI monetization scheme bites the dust.”
Our reading: Meta dropping a major GenAI project is one more GenAI monetization scheme that has failed.
No horizon Evidence as of
Limitation: Shutdown evidence comes from a reputable news report quoting Meta directly; no standalone Meta shutdown announcement was recovered.
Assessment: SupportedSaid
Observation
“Consequently, any “reasoning” or “planning” in LLMs is hit or miss, viable if the particulars of a given situation are close enough to what is in the training set, but highly vulnerable if they are not.”
Our reading: Marcus says any reasoning or planning in LLMs is hit or miss: viable if a situation's particulars are close enough to the training set, but highly vulnerable if they are not.
No horizon Evidence as of
Limitation: Reasoning models released after July 2024 are not assessed.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Observation
“Much as I admire the new pair of systems, they do have a serious shortcoming and one that is not unfamiliar. The systems both rely on a kind of cheat on the input: trained human coders translate all the Olympiad's input sentences into mathematical form.”
Our reading: AlphaProof and AlphaGeometry 2 rely on human coders translating Olympiad problems into formal form, a serious shortcoming.
No horizon Evidence as of
Limitation: The assessment concerns the July 2024 demonstration, not later systems.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Observation
“Nonetheless, GDM's new work is powerful proof of concept (pun intended!). Neural networks and symbol systems can be profitably combined, contra Hinton and LeCun.”
Our reading: Google DeepMind's work shows neural networks and symbolic systems can be profitably combined.
No horizon Evidence as of
Limitation: Technical usefulness does not imply financial profitability.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Observation
“Hallucinations and boneheaded errors of reasoning and what I like to call discomprehensions are, as far as I can tell, an inevitable byproduct of LLMs; certainly they have yet to disappear, despite many promises to the contrary.”
Our reading: Marcus says that, as far as he can tell, hallucinations, boneheaded reasoning errors and 'discomprehensions' are an inevitable byproduct of LLMs, and that they certainly have not yet disappeared despite many promises to the contrary.
No horizon Evidence as of
Limitation: Only the 'yet to disappear' part is scored.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Observation
“Long-term adoption (vs mere experimentation) has been modest, there is no significantly better tech now than there was in January; GPT-5 is still a no show.”
Our reading: GPT-5 had not been released as of July 2024.
No horizon Evidence as of
Limitation: This establishes public release status, not the state of private development.
Assessment: SupportedSaid
Causal claim
“As Ernie Davis and I pointed out in Rebooting AI, five years ago, part of the reason we are struggling with AI in complex AI systems is that we still lack adequate techniques for engineering complex systems. That hasn’t really changed.”
Our reading: Struggles with complex AI systems stem partly from a lack of adequate techniques for engineering complex systems, which has not really changed since 2019.
No horizon Evidence as of
Limitation: The causal assessment synthesizes technical explanations; it is not a controlled estimate of engineering's contribution to failures.
Source: Don’t look up: the massive Microsoft/Crowdstrike data outage is a huge wake-up call
Assessment: SupportedSaid
Forecast
“As of today, I am more confident than ever that GPT-5 won’t land this year.”
Our reading: OpenAI will not release GPT-5 in 2024.
Deadline passed Evidence as of
Limitation: This record assesses only the release timing of a model named GPT-5. It does not assess Marcus's reasons for the forecast or what GPT-5 could do.
Assessment: SupportedSaid
Forecast
“As of today, I am more confident than ever that GPT-5 won’t land this year.”
Our reading: GPT-5 will not be released in 2024.
Deadline passed Evidence as of
Limitation: This assesses the named public release, not private prototypes or differently named models.
Assessment: SupportedSaid
Observation
“Exhibit A: OpenAI’s CTO Mira Murati just publicly acknowledged what I long suspected: there is no mind blowing GPT-5 behind the scenes as of yet.”
Our reading: As of June 2024, OpenAI has no mind-blowing GPT-5 model waiting behind the scenes.
No horizon Evidence as of
Limitation: The decisive source is contemporaneous news reporting because the relevant primary interview transcript could not be recovered.
Assessment: SupportedSaid
Observation
“In many ways, it is much weaker than what Altman and I talked about at the Senate last year.”
Our reading: SB-1047 is in many ways much weaker than the AI regulation Marcus and Sam Altman discussed at the Senate in 2023.
No horizon Evidence as of
Limitation: The result depends on reading strength through the licensing and review mechanisms highlighted in the surrounding passage.
Assessment: SupportedSaid
Observation
“For one thing, the bad press about Sam Altman and OpenAI, who once seemingly could do no wrong, just keeps coming.”
Our reading: Negative press coverage of Sam Altman and OpenAI, once seen as able to do no wrong, keeps coming.
No horizon Evidence as of
Limitation: This is a finding about the existence of repeated coverage, not its prevalence or rate of increase.
Source: AI as Reality Television
Assessment: SupportedSaid
Observation
“For all the daily claims of “exponential progress”, and for all the tens of billions that have been invested since, reliability is still a dream, and of course toxicity and hallucinations are still with us.”
Our reading: Despite tens of billions of dollars invested since 2022, deep learning systems are still unreliable and still produce toxicity and hallucinations.
No horizon Evidence as of
Limitation: The reliability phrase is rhetorical and supplies no threshold.
Assessment: SupportedSaid
Observation
“OpenAI called itself open, and traded on the notion of being open, but even as early as May 2016 knew that the name was misleading.”
Our reading: OpenAI traded on being 'open' while knowing since May 2016 that the name was misleading.
No horizon Evidence as of
Limitation: The decisive May conversation comes from contemporaneous journalism, not a released internal email.
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: SupportedSaid
Observation
“Altman once (in 2016) promised that outsiders would play an important role in the company’s governance; that key promise has not been kept.”
Our reading: Altman's 2016 promise that outsiders would play an important role in OpenAI governance has not been kept.
No horizon Evidence as of
Limitation: The finding concerns public electoral participation, not the absence of independent directors.
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: SupportedSaid
Observation
“OpenAI has trained on a massive amount of copyrighted material, without consent, and in many instances without compensation.”
Our reading: OpenAI has trained on a massive amount of copyrighted material without consent and often without compensation.
No horizon Evidence as of
Limitation: The conclusion combines company disclosures with a rights holder’s filing; the filing is not a court finding.
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: SupportedSaid
Observation
“What the LLM does is more akin to what some high school students do when they plagiarize: change a few words here or there, while still sticking close to the original.”
Our reading: What the LLM in the glue-pizza example does is more akin to what some high school students do when they plagiarize: changing a few words while still sticking close to the original.
No horizon Evidence as of
Limitation: The supported scope is the supplied example, not LLM behavior generally.
Assessment: SupportedSaid
Observation
“Many people are reaching essential the same perceptions.”
Our reading: Many people are reaching essentially the same perceptions of Sam Altman.
No horizon Evidence as of
Limitation: The undefined word 'many' makes the result depend on a qualitative reading.
Source: Sam Altman’s fall from grace
Assessment: SupportedSaid
Observation
“In November, the narrative was about how everybody loved Sam. Things have changed. A lot.”
Our reading: The narrative about Sam Altman, which in November was that everybody loved him, has changed a lot.
No horizon Evidence as of
Limitation: The result depends on reading 'narrative' as prominent coverage and commentary, rather than measured population sentiment.
Source: Sam Altman’s fall from grace
Assessment: SupportedSaid
Observation
“I shared some of my own thoughts about which Pandora’s Boxes had (LLMs) and had not (AGI) already been opened.”
Our reading: The Pandora's Box of LLMs has already been opened.
No horizon Evidence as of
Limitation: Public availability does not establish the scale of malicious use or prove that all safeguards can be defeated.
Source: Jane Goodall, AI Skeptic
Assessment: SupportedSaid
Observation
“You might think that if OpenAI said, hey, we don’t want our software used for military purposes, Microsoft would respect that and say, ok cool, but they haven’t.”
Our reading: Microsoft has not respected OpenAI's stated wish that its software not be used for military purposes.
No horizon Evidence as of
Limitation: The result depends on reading respect as consistency with a public position, rather than contractual compliance.
Source: Microsoft and OpenAI’s increasingly complicated relationship
Assessment: SupportedSaid
Observation
“Playing the field with Mistral wasn’t enough; now he is building a really big model, big enough to compete directly with GPT-4, internally:”
Our reading: Beyond partnering with Mistral, Microsoft is building a large model in-house that is big enough to compete directly with GPT-4.
No horizon Evidence as of
Limitation: Support depends on reading compete as intended competitive positioning.
Source: Microsoft and OpenAI’s increasingly complicated relationship
Assessment: SupportedSaid
Observation
“You might think that after Satya offered to rescue Sam after Sam got fired, that Sam wouldn’t want to compete directly with Microsoft, but they are.”
Our reading: Sam Altman's OpenAI competes directly with Microsoft, even though Satya Nadella offered to rescue Altman after he was fired.
No horizon Evidence as of
Limitation: The customer meetings are documented by reputable reporting based on attendees, rather than public meeting records.
Source: Microsoft and OpenAI’s increasingly complicated relationship
Assessment: SupportedSaid
Forecast
“People who buy Altman’s narrative will continue to invest.”
Our reading: People who buy Altman's narrative will continue to invest.
Settled before its deadline Evidence as of
Limitation: Public endorsement is a proxy for belief.
Source: The Sam Altman Playbook
Assessment: SupportedSaid
Observation
“At the same talk at Stanford, Altman grandoisely asserted, with no qualification, that “we’re making AGI”, without touching on the possibility that current approaches might not get us there — despite (a) the enormous known problems with reliability, reasoning, planning and hallucinations and (b) the fact that the entire field, despite being massively well-capitalized, has struggled for over year (and in fact for decades) to solve any of these problems.”
Our reading: Current AI approaches have enormous known problems with reliability, reasoning, planning and hallucinations, and the entire field, despite being massively well-capitalized, has struggled for over a year, and in fact for decades, to solve any of these problems.
No horizon Evidence as of
Limitation: The conclusion depends on reading solve as general resolution, rather than improvement or success on bounded tasks.
Source: The Sam Altman Playbook
Assessment: SupportedSaid
Causal claim
“The very process that LLMs use to generalize also creates hallucinations.”
Our reading: The same process LLMs use to generalize is what creates their hallucinations.
No horizon Evidence as of
Limitation: The theory covers rare, arbitrary facts most clearly. Other kinds of hallucination may have other causes.
Assessment: SupportedSaid
Forecast
“LLMs are well on their way to becoming a commodity; price wars will be intense.”
Our reading: LLMs are becoming a commodity and price wars among providers will be intense.
Settled before its deadline Evidence as of
Limitation: Commoditization is assessed as a direction of travel, consistent with well on their way.
Source: Groupthink versus Tulips. What will historians think of 2024?
Assessment: SupportedSaid
Observation
“Adding visual data to LLMs hasn’t solved hallucinations.”
Our reading: Adding visual data to LLMs has not solved hallucinations.
No horizon Evidence as of
Limitation: 'Solved' is not defined. The claim is read as hallucinations remaining a material problem, not as adding vision bringing no improvement at all. The evidence packet notes that the GPT-4V system card reports fewer hallucinations during a beta, but that passage is not in the cited excerpt.
Source: TED, AI, and the test of time
Assessment: SupportedSaid
Observation
“I also urged the world to focus on global AI governance, and am pleased to see some genuine progress in that direction, especially led by the UN and the Council of Europe.”
Our reading: There has been genuine progress on global AI governance, led especially by the UN and the Council of Europe.
No horizon Evidence as of
Limitation: Institutional progress is distinct from demonstrated practical effectiveness.
Source: TED, AI, and the test of time