Claims checked against evidence, page 9 of 15
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Claims checked against evidence (1432)
Each of these claims was checked against dated evidence.
Page 9 of 15: claims 801 to 900 of 1432, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Gary Marcus Observation
“Most or all of the major big tech companies joined a lobbying organization that fought SB-1047, tooth and nail, despite broad public support for the bill.”
Our reading: Most or all major big tech companies joined a lobbying group that fought SB-1047 hard despite broad public support for it.
No horizon Evidence as of
Limitation: The majority assessment uses the contract's five-company group: Alphabet/Google, Amazon, Apple, Meta, and Microsoft.
Assessment: UnresolvedSaid
Gary Marcus Observation
“And no, GPT-5 did not drop this week as many had hoped.”
Our reading: GPT-5 was not released in the week before 2024-08-18, despite many people hoping it would be.
No horizon Evidence as of
Limitation: The target-period evidence does not fully cover August 12–18, 2024 [e1, e2].
Source: What has and has not changed in the AI since the ChatGPT revolution? [Video]
Assessment: SupportedSaid
Gary Marcus Observation
“Lastly, asking for standards is not unique to AI; it's common across many industries to ask that companies evaluate the safety of their products according to set standards: just look at the pharmaceutical industry, or aviation, automobiles, etc.”
Our reading: Asking for standards is not unique to AI: in many industries, such as pharmaceuticals, aviation and automobiles, it is common to ask companies to evaluate the safety of their products against set standards.
No horizon Evidence as of
Limitation: The evidence establishes the three named U.S. examples, not a numerical survey of every industry or jurisdiction.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047
Assessment: SupportedSaid
Gary Marcus Observation
“And much of what would be required is already something companies committed to voluntarily, in discussions at the White House and in Seoul.”
Our reading: Much of what SB-1047 would require is something AI companies already voluntarily committed to at the White House and in Seoul.
No horizon Evidence as of
Limitation: The assessment is a qualitative comparison, not a count of statutory clauses.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047
Assessment: SupportedSaid
Gary Marcus Observation
“OpenAI recently put Paul Nakasone (ex NSA) on the board”
Our reading: OpenAI recently put Paul Nakasone, formerly of the NSA, on its board.
No horizon Evidence as of
Limitation: This establishes the appointment and career background, not any implication about surveillance.
Source: OpenAI: On a Path to Becoming The World’s Most Frightening Surveillance Company?
Assessment: UnresolvedSaid
Gary Marcus Observation
“Sam Altman has acknowledged wanting to train on everyone's personal documents (Word files, email etc)”
Our reading: Sam Altman has acknowledged that he wants to train AI on everyone's personal documents, such as Word files and email.
No horizon Evidence as of
Limitation: The search cannot establish that no other qualifying statement exists.
Source: OpenAI: On a Path to Becoming The World’s Most Frightening Surveillance Company?
Assessment: UnresolvedSaid
Gary Marcus Observation
“OpenAI just put in money in a $60M fundraise with a Webcam company and is planning hardware joint venture with them.”
Our reading: OpenAI recently invested in a $60M fundraise for a webcam company and is planning a hardware joint venture with it.
No horizon Evidence as of
Limitation: The original financial report could not be read with read_page.
Source: OpenAI: On a Path to Becoming The World’s Most Frightening Surveillance Company?
Assessment: SupportedSaid
Gary Marcus Observation
“An awful lot of key players have left the island. Most voluntarily.”
Our reading: Many key OpenAI people have left, most of them voluntarily.
No horizon Evidence as of
Limitation: The roster is a transparent group drawn from contemporaneous reporting, not a complete personnel register.
Source: The OpenAI Plot Thickens
Assessment: UnresolvedSaid
Gary Marcus Forecast
“LLMs are not AGI, and (on their own) never will be; scaling alone was never going to be enough.”
Our reading: LLMs on their own are not and will never be AGI.
No deadline stated Evidence as of
Limitation: The review cannot establish the absence of every possible demonstration.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“LLMs are not AGI, and (on their own) never will be; scaling alone was never going to be enough.”
Our reading: Scaling alone was never going to be enough to reach AGI.
No deadline stated Evidence as of
Limitation: Benchmark correlations do not establish whether scaling alone is sufficient for AGI [e1].
Assessment: SupportedSaid
Gary Marcus Observation
“Nvidia dropped 6%, 20% over the last month”
Our reading: Nvidia stock fell 6% on August 5, 2024, and 20% over the previous month.
No horizon Evidence as of
Limitation: The result depends on reading the figures as rounded closing-price changes.
Assessment: SupportedSaid
Gary Marcus Observation
“OpenAI tempered expectations for its next event, and said we wouldn’t see GPT-5 then.”
Our reading: OpenAI tempered expectations for its next event and said we would not see GPT-5 at that event.
No horizon Evidence as of
Limitation: Evidence is a reporter's direct quotation, not an accessible original company announcement.
Assessment: UnresolvedSaid
Gary Marcus Observation
“The bubble has begun to burst.”
Our reading: As of late July 2024, the GenAI bubble has begun to burst.
No horizon Evidence as of
Limitation: A narrower claim about a technology-stock correction would be easier to support.
Assessment: UnresolvedSaid
Gary Marcus Observation
“I have long argued that Generative AI might be a dud; I just didn’t expect it to fade so fast.”
Our reading: Generative AI is fading fast, as of July 2024, and may turn out to be a dud.
No horizon Evidence as of
Limitation: The broad wording requires an evaluator-supplied reading.
Assessment: SupportedSaid
Gary Marcus Observation
“Another GenAI monetization scheme bites the dust.”
Our reading: Meta dropping a major GenAI project is one more GenAI monetization scheme that has failed.
No horizon Evidence as of
Limitation: Shutdown evidence comes from a reputable news report quoting Meta directly; no standalone Meta shutdown announcement was recovered.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Users have lost faith, clients have lost faith, VC’s have lost faith.”
Our reading: As of July 2024, users, clients, and venture capitalists have lost faith in generative AI.
No horizon Evidence as of
Limitation: General-public opinions cannot be treated as a representative survey of GenAI users.
Assessment: UnresolvedSaid
Gary Marcus Observation
“PS Microsoft Bing completely flubbed, 4/4, “people going yuck towards Generative AI”.”
Our reading: Microsoft Bing completely flubbed the prompt "people going yuck towards Generative AI", 4 out of 4 results.
No horizon Evidence as of
Limitation: Only one search query was run, so an independent record could still exist that this search missed.
Source: Most people don’t think GenAI has improved their productivity
Assessment: SupportedSaid
Gary Marcus Observation
“Consequently, any “reasoning” or “planning” in LLMs is hit or miss, viable if the particulars of a given situation are close enough to what is in the training set, but highly vulnerable if they are not.”
Our reading: Marcus says any reasoning or planning in LLMs is hit or miss: viable if a situation's particulars are close enough to the training set, but highly vulnerable if they are not.
No horizon Evidence as of
Limitation: Reasoning models released after July 2024 are not assessed.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Gary Marcus Observation
“People are starting to realize all of the above.”
Our reading: People are starting to realize generative AI has plateaued, lacks a killer app, still hallucinates and errs, and has no moat.
No horizon Evidence as of
Limitation: The population and size of the claimed change are unspecified.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Gary Marcus Observation
“Much as I admire the new pair of systems, they do have a serious shortcoming and one that is not unfamiliar. The systems both rely on a kind of cheat on the input: trained human coders translate all the Olympiad's input sentences into mathematical form.”
Our reading: AlphaProof and AlphaGeometry 2 rely on human coders translating Olympiad problems into formal form, a serious shortcoming.
No horizon Evidence as of
Limitation: The assessment concerns the July 2024 demonstration, not later systems.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“That said, I am certain that the impending collapse won’t lead to the absolute death of AI. There is too much at stake.”
Our reading: The impending generative AI collapse will not lead to the absolute death of AI.
No deadline stated Evidence as of
Only if: after the generative AI collapse
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: ContradictedSaid
Gary Marcus Forecast
“I fully expect that the generative AI bubble will begin to burst within the next 12 months, for many reasons:”
Our reading: The generative AI bubble will begin to burst by July 28, 2025.
Deadline passed Evidence as of
Limitation: The result depends on interpreting burst as a broad, sustained unwinding.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Gary Marcus Forecast
“There can be no path to AGI without neurosymbolic AI.”
Our reading: AGI cannot be achieved without neurosymbolic AI.
No deadline stated Evidence as of
Limitation: Public demonstrations may not disclose enough implementation detail to classify a system's methods.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Gary Marcus Observation
“Nobody has a moat”
Our reading: No generative AI company has a competitive moat.
No horizon Evidence as of
Limitation: The universal claim requires unusually broad evidence to support.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Gary Marcus Observation
“Nonetheless, GDM's new work is powerful proof of concept (pun intended!). Neural networks and symbol systems can be profitably combined, contra Hinton and LeCun.”
Our reading: Google DeepMind's work shows neural networks and symbolic systems can be profitably combined.
No horizon Evidence as of
Limitation: Technical usefulness does not imply financial profitability.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Gary Marcus Forecast
“The neural network approach known as generative AI — especially its poster chatbot ChatGPT – has captured the world’s imagination, but will soon flame out.”
Our reading: Generative AI, especially ChatGPT, will soon flame out.
No deadline stated Evidence as of
Limitation: OpenAI's disclosure is self-reported.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Gary Marcus Observation
“Hallucinations and boneheaded errors of reasoning and what I like to call discomprehensions are, as far as I can tell, an inevitable byproduct of LLMs; certainly they have yet to disappear, despite many promises to the contrary.”
Our reading: Marcus says that, as far as he can tell, hallucinations, boneheaded reasoning errors and 'discomprehensions' are an inevitable byproduct of LLMs, and that they certainly have not yet disappeared despite many promises to the contrary.
No horizon Evidence as of
Limitation: Only the 'yet to disappear' part is scored.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: ContradictedSaid
Gary Marcus Observation
“The current approach has reached a plateau”
Our reading: The current generative AI approach has reached a plateau.
No horizon Evidence as of
Limitation: Arena preferences can reflect presentation as well as correctness.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Gary Marcus Observation
“So for the most part, we’ve been stuck in a world in which all the major tech companies are building exactly the same thing (bigger and bigger LLMs based on more data), achieving almost exactly the same results (a passel of GPT-4 level models, hardly different from each other, all struggling with hallucinations and boneheaded errors), and hardly any investment in anything else.”
Our reading: Marcus says that, for the most part, all the major tech companies are building the same thing (bigger LLMs on more data) and getting almost the same results (GPT-4-level models that barely differ and all struggle with hallucinations and boneheaded errors), with hardly any investment in anything else.
No horizon Evidence as of
Limitation: Company evaluations are not a substitute for consistent independent comparisons.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Gary Marcus Observation
“There is no killer app”
Our reading: Generative AI has no killer app.
No horizon Evidence as of
Limitation: Killer app remains a qualitative threshold.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Gary Marcus Forecast
“The biggest intrinsic failings of generative AI have to do with reliability, in a way that I believe can never be solved, given their inherent nature.”
Our reading: Generative AI's reliability failings can never be solved, given its inherent nature.
No deadline stated Evidence as of
Limitation: The cited empirical study covers two benchmarks and a finite set of agents.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Customers can easily switch to other providers, almost anytime they want, and for month there had been more and more alternatives, including Anthropic and Google, at more and more attractive prices, forcing OpenAI itself to cut prices just a few days ago.”
Our reading: Customers can easily switch LLM providers, and growing cheaper alternatives from Anthropic and Google forced OpenAI to cut prices in July 2024.
No horizon Evidence as of
Limitation: Company launch claims are not independent price-performance measurements.
Assessment: UnresolvedSaid
Gary Marcus Observation
“LLMs have just became exactly the commodity I predicted they would become, at the lowest possible price.”
Our reading: LLMs have become a commodity available for free.
No horizon Evidence as of
Limitation: Capability comparisons in the cited primary sources come from Meta.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Obviously, their only hope is to raise more cash, and they will certainly try.”
Our reading: OpenAI's only hope of survival is raising more cash, and it will try to.
No deadline stated Evidence as of
Limitation: This verdict retains both parts of the supplied claim; it does not silently reduce it to the fundraising prediction.
Assessment: UnresolvedSaid
Gary Marcus Observation
“And leaving more and more code writing to generative AI, which grasps syntax but not meaning, as a new benchmark from Stephen Wolfram shows, is not the answer, either.”
Our reading: Generative AI grasps syntax but not meaning, as shown by a new Stephen Wolfram benchmark.
No horizon Evidence as of
Limitation: This is an evidence gap, not a finding that Marcus misreported the benchmark.
Source: Don’t look up: the massive Microsoft/Crowdstrike data outage is a huge wake-up call
Assessment: SupportedSaid
Gary Marcus Causal claim
“As Ernie Davis and I pointed out in Rebooting AI, five years ago, part of the reason we are struggling with AI in complex AI systems is that we still lack adequate techniques for engineering complex systems. That hasn’t really changed.”
Our reading: Struggles with complex AI systems stem partly from a lack of adequate techniques for engineering complex systems, which has not really changed since 2019.
No horizon Evidence as of
Limitation: The causal assessment synthesizes technical explanations; it is not a controlled estimate of engineering's contribution to failures.
Source: Don’t look up: the massive Microsoft/Crowdstrike data outage is a huge wake-up call
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Neurosymbolic AI has long been an underdog; in the end I expect it to come from behind and be essential.”
Our reading: Neurosymbolic AI will eventually come from behind and become essential.
No deadline stated Evidence as of
Limitation: The intended destination and meaning of essential require interpretation.
Source: Marcus goes gaga over Gates clip
Assessment: SupportedSaid
Gary Marcus Forecast
“As of today, I am more confident than ever that GPT-5 won’t land this year.”
Our reading: OpenAI will not release GPT-5 in 2024.
Deadline passed Evidence as of
Limitation: This record assesses only the release timing of a model named GPT-5. It does not assess Marcus's reasons for the forecast or what GPT-5 could do.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Others are bailing now, too. Or at least trying to subtly alter their positions, committing less to the unrealistic.”
Our reading: Prominent AI figures are now retreating from, or quietly softening, their unrealistic claims about LLMs and AGI.
No horizon Evidence as of
Limitation: Restricting others solely to the two named examples would produce a narrower assessment than the wording clearly warrants.
Assessment: SupportedSaid
Gary Marcus Observation
“Exhibit A: OpenAI’s CTO Mira Murati just publicly acknowledged what I long suspected: there is no mind blowing GPT-5 behind the scenes as of yet.”
Our reading: As of June 2024, OpenAI has no mind-blowing GPT-5 model waiting behind the scenes.
No horizon Evidence as of
Limitation: The decisive source is contemporaneous news reporting because the relevant primary interview transcript could not be recovered.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“LLMs are not AGI, and (on their own) never will be; scaling alone was never going to be enough.”
Our reading: Scaling alone is not enough to reach AGI.
No horizon Evidence as of
Limitation: Finite scaling experiments cannot establish what every larger system could achieve.
Assessment: SupportedSaid
Gary Marcus Observation
“In many ways, it is much weaker than what Altman and I talked about at the Senate last year.”
Our reading: SB-1047 is in many ways much weaker than the AI regulation Marcus and Sam Altman discussed at the Senate in 2023.
No horizon Evidence as of
Limitation: The result depends on reading strength through the licensing and review mechanisms highlighted in the surrounding passage.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Anyone who can afford that can easily fill out some paperwork and run a few tests, just like I can pay a few bucks to get a driver’s license. It’s the least they can do.”
Our reading: Companies that can afford $100M+ training runs can easily afford SB-1047's paperwork and testing requirements.
No horizon Evidence as of
Limitation: The packet contains no documented estimate of the June proposal's incremental paperwork and testing costs matched to developers' available resources and existing commitments.
Assessment: ContradictedSaid
Gary Marcus Observation
“He doesn’t specify which actual restrictions in the proposal bill would “stifle AI models”.”
Our reading: Andrew Ng's critique does not identify which specific SB-1047 provisions would stifle AI model builders.
No horizon Evidence as of
Limitation: This assessment does not validate Ng's legal interpretation or his prediction about innovation.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Every company that would be subject to the stronger set of regulations is valued in the billions.”
Our reading: Every company that would be subject to SB-1047's stronger regulations is valued in the billions of dollars.
No horizon Evidence as of
Limitation: Private-company valuations are intermittent and may refer to different share classes or transaction terms.
Assessment: UnresolvedSaid
Gary Marcus Observation
“It can do some things better, some worse, very few reliably.”
Our reading: Compared with a smart high schooler, GPT-4 can do some things better and some things worse, and it can do very few things reliably.
No horizon Evidence as of
Limitation: Both sources are company disclosures from 2023 and do not establish the performance of an identified June 2024 GPT-4 version [e1][e2].
Source: AGI by 2027?
Assessment: UnresolvedSaid
Gary Marcus Observation
“The double Y-axis makes no sense, and presupposes its own conclusion.”
Our reading: The double Y-axis on Aschenbrenner's graph makes no sense and presupposes its own conclusion.
No horizon Evidence as of
Limitation: The phrase makes no sense remains an evaluative judgment.
Source: AGI by 2027?
Assessment: MixedSaid
Gary Marcus Observation
“The graph assumes that all continued progress will be same — ignoring qualitatively unsolved problems (hallucinations, planning, reasoning etc), the fact that synthetic data may not be as useful as natural data, limits on then-possible energy distribution and compute, etc.”
Our reading: Aschenbrenner's graph assumes all continued progress will be the same, ignoring qualitatively unsolved problems such as hallucinations, planning and reasoning, the possibility that synthetic data may not be as useful as natural data, and limits on energy distribution and compute.
No horizon Evidence as of
Limitation: The result depends on including the graph's supporting analysis.
Source: AGI by 2027?
Assessment: SupportedSaid
Gary Marcus Observation
“For one thing, the bad press about Sam Altman and OpenAI, who once seemingly could do no wrong, just keeps coming.”
Our reading: Negative press coverage of Sam Altman and OpenAI, once seen as able to do no wrong, keeps coming.
No horizon Evidence as of
Limitation: This is a finding about the existence of repeated coverage, not its prevalence or rate of increase.
Source: AI as Reality Television
Assessment: UnresolvedSaid
Gary Marcus Observation
“In 2019 I pointed out through a series of examples on Twitter with GPT-2 that LLMs were subject to essentially the same critique: greedy with respect to data, uninterpretable, unreliable as reasoners, subject to hallucinations, etc, that they lacked world models (“they don't develop robust representations of *how events unfold over time*), and that they would be inadequate on their own.”
Our reading: In 2019, using a series of GPT-2 examples on Twitter, Marcus pointed out that LLMs were greedy with respect to data, uninterpretable, unreliable as reasoners, and subject to hallucinations, that they lacked world models, and that they would be inadequate on their own.
No horizon Evidence as of
Limitation: The reply does not document the boundaries or full contents of the claimed 2019 GPT-2 Twitter series [e1].
Assessment: UnresolvedSaid
Gary Marcus Observation
“Nowadays we both (unlike many other people) stress the absolutely essential nature of common sense, physical reasoning and world models, and the failure of current architectures to handle those well.”
Our reading: Marcus and LeCun, unlike many other people, now both stress that common sense, physical reasoning and world models are absolutely essential, and that current architectures fail to handle them well.
No horizon Evidence as of
Limitation: Public agreement does not establish who originated an idea or whether one speaker changed his position.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Worse, we are no closer to a solution to the alignment problem now than we were then.”
Our reading: There has been no progress toward solving the AI alignment problem between 2022 and mid-2024.
No horizon Evidence as of
Limitation: A narrower interpretation covering any improvement in present-model alignment would favor contradiction.
Assessment: SupportedSaid
Gary Marcus Observation
“For all the daily claims of “exponential progress”, and for all the tens of billions that have been invested since, reliability is still a dream, and of course toxicity and hallucinations are still with us.”
Our reading: Despite tens of billions of dollars invested since 2022, deep learning systems are still unreliable and still produce toxicity and hallucinations.
No horizon Evidence as of
Limitation: The reliability phrase is rhetorical and supplies no threshold.
Assessment: UnresolvedSaid
Gary Marcus Observation
“But OpenAI has been and continues to be a California-registered 501(3)(c) nonprofit, pledged not to profits, but to humanity – and still getting tax-exempt status on the basis of their promise to deploy AI safely:”
Our reading: OpenAI is a registered nonprofit receiving tax-exempt status based on its promise to deploy AI safely.
No horizon Evidence as of
Limitation: The packet lacks dated records confirming continued California registration and federal exemption through May 30, 2024; this does not establish that either had lapsed.
Source: Disturbing and misleading efforts to defame Helen Toner through misdirection
Assessment: UnresolvedSaid
Gary Marcus Observation
“The 3.5 API (geared towards programmers as a backend tool) was available beforehand, but that’s not of course the consumer-directed product with gimmicks like word-by-word typing to give a false impression of humanness that ultimately went viral.”
Our reading: ChatGPT, unlike the earlier GPT-3.5 API, was a consumer product using gimmicks like word-by-word typing to give a false impression of humanness.
No horizon Evidence as of
Limitation: Earlier model availability rests on a contemporaneous community report, which does not identify an official product named GPT-3.5 API [e2].
Source: Disturbing and misleading efforts to defame Helen Toner through misdirection
Assessment: UnresolvedSaid
Gary Marcus Observation
“Fundamentally, the essay attacks a strawman, viz. that the firing of Altman was somehow precipitated by an immediate safety concern), thereby diverting from the board’s actual, stated concern (candor).”
Our reading: The Taylor/Summers essay attacks a strawman about safety, diverting from the board's actual concern of candor in firing Altman.
No horizon Evidence as of
Limitation: The complete essay was unavailable; the retrieved reporting cannot establish what the full text omitted.
Source: Disturbing and misleading efforts to defame Helen Toner through misdirection
Assessment: UnresolvedSaid
Gary Marcus Observation
“And oops, Grok’s basic understanding of time is shaky, too:”
Our reading: Grok has a shaky basic understanding of time.
No horizon Evidence as of
Limitation: A missing public test is not evidence that Grok performed well.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Musk is just building bigger and bigger LLMs, and Meta is mostly (despite LeCun’s reservations) mostly doing the same.”
Our reading: Musk's xAI is just building bigger LLMs, and Meta is mostly doing the same.
No horizon Evidence as of
Limitation: Public announcements do not provide a complete account of internal research.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“By clustering piles of similar things together in a giant n-dimensional space, they pretty much automatically become synonym and pastische machines, regurgitating a lot of words with slight paraphrases while adding conceptually little, and understanding even less.”
Our reading: Because LLMs cluster similar things in a high-dimensional space, they become synonym and pastiche machines that regurgitate with slight paraphrase, adding little and understanding less.
No horizon Evidence as of
Limitation: Verbatim recall and memorization from similar sequences do not establish how often models produce close paraphrases in ordinary use [e1, e3].
Assessment: SupportedSaid
Gary Marcus Observation
“Many people are reaching essential the same perceptions.”
Our reading: Many people are reaching essentially the same perceptions of Sam Altman.
No horizon Evidence as of
Limitation: The undefined word 'many' makes the result depend on a qualitative reading.
Source: Sam Altman’s fall from grace
Assessment: SupportedSaid
Gary Marcus Observation
“In November, the narrative was about how everybody loved Sam. Things have changed. A lot.”
Our reading: The narrative about Sam Altman, which in November was that everybody loved him, has changed a lot.
No horizon Evidence as of
Limitation: The result depends on reading 'narrative' as prominent coverage and commentary, rather than measured population sentiment.
Source: Sam Altman’s fall from grace
Assessment: UnresolvedSaid
Gary Marcus Observation
“It’s a pattern.”
Our reading: Sam Altman's lack of candor, seen in his statements to the Senate about OpenAI equity, his lies about Helen Toner and his lies about Scarlett Johansson, is a pattern.
No horizon Evidence as of
Limitation: The equity evidence does not establish whether the Senate answer was a misleading omission or material deception.
Source: The OpenAI board was right
Assessment: UnresolvedSaid
Gary Marcus Observation
“I probably don’t have to tell you, but that’s complete bullshit. And stupid, obviously refuted bullshit at that.”
Our reading: OpenAI's claim that its voice's resemblance to Scarlett Johansson was coincidental is false.
No horizon Evidence as of
Limitation: Earlier casting evidence is context outside the denial's target period.
Source: The OpenAI board was right
Assessment: UnresolvedSaid
Gary Marcus Observation
“They said it wasn’t intentional, but of course it was.”
Our reading: OpenAI intentionally made its voice resemble Scarlett Johansson's.
No horizon Evidence as of
Limitation: The primary production account comes from the company whose conduct is disputed.
Source: The OpenAI board was right
Assessment: UnresolvedSaid
Gary Marcus Observation
“I can’t tell you what happened to Sam’s caps lock, but obviously the claim that the resemblance to ScarJo was a “coincidence” was a lie. Sam knew perfectly well what the character sounded like.”
Our reading: Sam Altman knew the voice resembled Johansson's Her character, so the coincidence claim was a lie.
No horizon Evidence as of
Limitation: The public record is largely circumstantial and includes interested-party accounts.
Source: The OpenAI board was right
Assessment: UnresolvedSaid
Gary Marcus Observation
“Lots of quirky errors are already being reported, same as ever, like this reasoning error from Jane Rosenzweig:”
Our reading: Lots of quirky errors in GPT-4o are already being reported, same as ever, such as a reasoning error reported by Jane Rosenzweig.
No horizon Evidence as of
Limitation: Social-media text extraction omitted the underlying example.
Source: Hot take on OpenAI’s new GPT-4o
Assessment: UnresolvedSaid
Gary Marcus Observation
“They don’t have GPT-5 after 14 months of trying.”
Our reading: OpenAI does not have GPT-5 after 14 months of trying since GPT-4.
No horizon Evidence as of
Limitation: Internal completion status may not be public.
Source: Hot take on OpenAI’s new GPT-4o
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“LLMs confined to X tweets will be even less grounded in reality.”
Our reading: LLMs whose synthesis is confined to X tweets will be even less grounded in reality than LLMs in general.
No horizon Evidence as of
Limitation: This is an unresolved causal assessment, not a finding that Marcus was wrong.
Assessment: SupportedSaid
Gary Marcus Causal claim
“The very process that LLMs use to generalize also creates hallucinations.”
Our reading: The same process LLMs use to generalize is what creates their hallucinations.
No horizon Evidence as of
Limitation: The theory covers rare, arbitrary facts most clearly. Other kinds of hallucination may have other causes.
Assessment: MixedSaid
Gary Marcus Observation
“Instead, everything that LLMs say -- true or false -- comes from the same process of statistically reconstructing what words are likely in some context. They NEVER fact-check what they say.”
Our reading: All LLM output, true or false, comes from the same statistical word-reconstruction process, and LLMs never fact-check what they say.
No horizon Evidence as of
Limitation: The packet documents next-token generation only for GPT-4 and does not cover every LLM in use during the period.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“That means, of course, that OpenAI’s business model is threatened; same for Anthropic’s, Mistral’s, and so on.”
Our reading: Meta's free near-GPT-4 model threatens the business models of OpenAI, Anthropic, Mistral and similar companies.
No horizon Evidence as of
Limitation: Commercial threat is broader than benchmark performance.
Source: Groupthink versus Tulips. What will historians think of 2024?
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Whatever tiny moat there was between the companies will further diminish.”
Our reading: The small moat between LLM companies will further diminish.
No deadline stated Evidence as of
Limitation: The ordinary business meaning of moat is broader than model quality.
Source: Groupthink versus Tulips. What will historians think of 2024?
Assessment: SupportedSaid
Gary Marcus Forecast
“LLMs are well on their way to becoming a commodity; price wars will be intense.”
Our reading: LLMs are becoming a commodity and price wars among providers will be intense.
Settled before its deadline Evidence as of
Limitation: Commoditization is assessed as a direction of travel, consistent with well on their way.
Source: Groupthink versus Tulips. What will historians think of 2024?
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“A billion robots with a superficial understanding of the physical world would be a recipe for disaster.”
Our reading: Deploying a billion robots with only superficial understanding of the physical world would lead to disaster.
No deadline stated Evidence as of
Only if: if robots have only superficial understanding of the physical world
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Khosla is a venture capitalist, and I think venture capitalists will have a huge role in whether or not we get to the promised land.”
Our reading: Marcus thinks venture capitalists will have a huge role in whether or not we get to the promised land of world-improving technology.
No deadline stated Evidence as of
Limitation: The destination and magnitude of influence remain qualitative.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Microsoft’s Mustafa Suleyman promised that hallucinations would be cured “soon”, yet my X feed is still filled with examples like these from Princeton Professor Aleksandra Korolova:”
Our reading: Although Mustafa Suleyman promised that hallucinations would be cured "soon", Marcus's X feed was still filled with examples of AI hallucinations, such as those from Princeton Professor Aleksandra Korolova.
No horizon Evidence as of
Limitation: Partial support is not a mixed outcome because no contradictory outcome evidence was found.
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“We won’t get to a billion personal robots if they are as dodgy as driverless cars, frequently working, yet stymied often enough by outliers that we can’t fully count on them.”
Our reading: A billion personal robots will not be deployed if robots are as unreliable with outliers as driverless cars are.
No deadline stated Evidence as of
Only if: if robots are as unreliable as driverless cars
Assessment: SupportedSaid
Gary Marcus Observation
“Adding visual data to LLMs hasn’t solved hallucinations.”
Our reading: Adding visual data to LLMs has not solved hallucinations.
No horizon Evidence as of
Limitation: 'Solved' is not defined. The claim is read as hallucinations remaining a material problem, not as adding vision bringing no improvement at all. The evidence packet notes that the GPT-4V system card reports fewer hallucinations during a beta, but that passage is not in the cited excerpt.
Source: TED, AI, and the test of time
Assessment: UnresolvedSaid
Gary Marcus Observation
“I think it’s fair to say that the multimodal stuff still is very buggy, having troubles with basics like the English alphabet.”
Our reading: OpenAI's multimodal capabilities remain very buggy as of April 2024, struggling even with basics like the English alphabet.
No horizon Evidence as of
Limitation: The assessment is limited by source-access failures.
Source: TED, AI, and the test of time
Assessment: UnresolvedSaid
Gary Marcus Observation
“I warned that LLMs would be used by bad actors, and they increasingly are.”
Our reading: LLMs are increasingly being used by bad actors.
No horizon Evidence as of
Limitation: Malicious use can remain undetected.
Source: TED, AI, and the test of time
Assessment: UnresolvedSaid
Gary Marcus Observation
“I am just not seeing that as OpenAI’s mission, anymore, especially when they have continued to fail to compensate most of the creators whose work they are drawing on, largely without consent.”
Our reading: OpenAI no longer appears to pursue its mission of ensuring AGI benefits all of humanity, and it has continued not to compensate most creators whose work it uses, largely without their consent.
No horizon Evidence as of
Limitation: The factual component is assessable in principle, but the required population and coverage records are missing.
Source: TED, AI, and the test of time
Assessment: MixedSaid
Gary Marcus Observation
“The study Mollick linked to doesn’t actually show what he claims. If you read it carefully, it says literally nothing about capabilities improving.”
Our reading: The study Mollick cited does not show that LLM capabilities are improving; it only addresses compute efficiency.
No horizon Evidence as of
Limitation: Language-model performance is narrower than general capability.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Gary Marcus Observation
“What it actually shows is some improvements, mostly modest, over the last few months for a bunch of different measures.”
Our reading: OpenAI's GPT-4 Turbo graph shows only mostly modest improvements over recent months.
No horizon Evidence as of
Limitation: Modest is a qualitative judgment, and relative changes can look larger.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: UnresolvedSaid
Gary Marcus Observation
“Another way to think about this (see third graph above) is that there about 5 - 7 recent models are on a par with GPT-4, but none is clearly and decisively ahead.”
Our reading: About 5 to 7 recent models are on par with GPT-4, but none is clearly ahead.
No horizon Evidence as of
Limitation: The earlier four-model list does not contradict a later approximate five-to-seven-model count.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Gary Marcus Observation
“I think it’s fair to say that they have not. GPT-Turbo released this week still produces its share of groaners, like this interchange Phil Libin just sent:”
Our reading: Hallucinations and stupid errors in LLMs have not been solved; GPT-4 Turbo still makes them.
No horizon Evidence as of
Limitation: The coding test measures task completion, not hallucination frequency.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: UnresolvedSaid
Gary Marcus Observation
“One more way to look at this, is this graph I just saw: enormous convergence on GPT-4 level performance, in multiple models released since, yet nothing decisively ahead.”
Our reading: Multiple models released since GPT-4 have converged on GPT-4-level performance with none decisively ahead.
No horizon Evidence as of
Limitation: A coding benchmark cannot establish general capability parity.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“If enthusiasm for GenAI dwindles and market valuations plummet, AI won’t disappear, and LLMs won’t disappear; they will still have their place as tools for statistical approximation.”
Our reading: If enthusiasm for GenAI dwindles and market valuations plummet, AI and LLMs will not disappear, and LLMs will still have their place as tools for statistical approximation.
No deadline stated Evidence as of
Only if: if enthusiasm for GenAI dwindles and market valuations plummet
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Gary Marcus Observation
“Already in recent weeks, Inflection AI largely closed shop, Stability AI is struggling, and LLM-based autonomous vehicle company called Ghost closed shop, and a software engineer on YouTube raised quite serious questions about the wildly-hyped AI coding system Devin.”
Our reading: In recent weeks Inflection AI largely closed, Stability AI is struggling, Ghost closed, and serious questions were raised about Devin.
No horizon Evidence as of
Limitation: The Inflection finding depends on the meaning of 'largely'; the company continued operating.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Gary Marcus Observation
“Huge jump from GPT-2 to GPT-3. Huge jump from GPT-3 to GPT-4 … and not so much for GPT-4 (13 months ago) to GPT-4 Turbo (just released). It’s hard not to see this plot as tentative evidence for the hypothesis of diminishing returns.”
Our reading: MMLU gains from GPT-4 to GPT-4 Turbo are small compared with earlier generational jumps, tentatively evidencing diminishing returns.
No horizon Evidence as of
Limitation: The records do not compare all four models under one protocol: GPT-2 was fine-tuned, GPT-3 used few-shot prompting, GPT-4 used five-shot prompting, and Turbo used zero-shot chain-of-thought prompting [e1, e2, e3].
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: UnresolvedSaid
Gary Marcus Forecast
“I couldn’t agree more. And parallel arguments could be made in many domains.”
Our reading: Marcus endorses the view that ChatGPT and similar tools will not themselves be virtual scientists anytime soon, and says parallel arguments could be made in many domains.
No deadline stated Evidence as of
Limitation: “Anytime soon” supplies no deadline; reaching the evaluation date cannot establish support.
Source: Superhuman AGI is not nigh
Assessment: SupportedSaid
Gary Marcus Observation
“Tesla FSD is getting better. But is getting enough better?”
Our reading: Tesla FSD is getting better, and he raises the question of whether it is getting enough better.
No horizon Evidence as of
Limitation: The tracker figures in e1 come from crowdsourced, self-reported data. e1 notes that data for later v12.3.x releases still needed weeks to be entered.
Source: 24 hours in AI news
Assessment: SupportedSaid
Gary Marcus Observation
“current models aren’t literally look up tables; they can generalize to some degree, but not enough.”
Our reading: Current AI models can generalize to some degree but not enough.
No horizon Evidence as of
Limitation: The outcome depends on interpreting 'not enough' through the surrounding discussion of reliability.
Assessment: ContradictedSaid
Gary Marcus Causal claim
“This is why driverless cars are still just demos, and why LLMs will never be reliable.”
Our reading: Poor generalization beyond training data is why driverless cars are still just demos.
No horizon Evidence as of
Limitation: The decisive operational account is the company's regulatory filing.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“It also drives their greediness with data, and even the ethical choices their developers have been making.”
Our reading: Neural networks' trouble generalizing beyond their training space, at least in the configurations dominant over the last three decades, also drives their greediness with data and even the ethical choices their developers have been making.
No horizon Evidence as of
Limitation: The technical evidence concerns particular recurrent networks and tasks, and its link to data demand is tentative [e1].
Assessment: SupportedSaid
Gary Marcus Observation
“Neural networks (at least in the configurations that have been dominant over the last three decades) have trouble generalizing beyond the multidimensional space that surrounds their training examples.”
Our reading: Neural networks in the configurations dominant for the last three decades have trouble generalizing beyond the space surrounding their training examples.
No horizon Evidence as of
Limitation: The outcome depends on reading the spatial language as unfamiliar conditions rather than a precise geometric boundary.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“In short, diminishing returns are what we can expect.”
Our reading: Scaling data for current AI models will yield diminishing returns.
No deadline stated Evidence as of
Limitation: A narrower claim about a specified loss curve could be supported by individual experiments.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Sooner or later, the exponential greed for data will exceed what is available.”
Our reading: The exponentially growing demand for training data will eventually exceed the data available.
No deadline stated Evidence as of
Limitation: Epoch's quantitative estimate covers public human-generated text, not all usable training data [e1].
Assessment: UnresolvedSaid
Gary Marcus Observation
“an intellectual monoculture — with no alternative approach that is nearly as well-funded.”
Our reading: The AI field is an intellectual monoculture in which no alternative approach is nearly as well-funded as generative AI.
No horizon Evidence as of
Limitation: The annual reporting period is inferred.
Assessment: SupportedSaid
Gary Marcus Observation
“The good news is that word is finally getting out more broadly.”
Our reading: Awareness of the danger generative AI poses to the information ecosystem is finally spreading more widely.
No horizon Evidence as of
Limitation: The trend ('more broadly') is inferred from high levels, not from a measured before-and-after comparison.
Assessment: SupportedSaid
Gary Marcus Observation
“Meanwhile X itself is promoting nonsense.”
Our reading: X itself is promoting nonsense.
No horizon Evidence as of
Limitation: The assessment relies on direct news reporting with screenshots and an independent fact-check; no separately retrievable original X display was obtained.