Claims checked against evidence by Gary Marcus, page 9 of 15
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Claims checked against evidence by Gary Marcus (1403)
Each of these claims was checked against dated evidence.
Page 9 of 15: claims 801 to 900 of 1403, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: UnresolvedSaid
Observation
“There is no killer app”
Our reading: Generative AI has no killer app.
No horizon Evidence as of
Limitation: Killer app remains a qualitative threshold.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Forecast
“The biggest intrinsic failings of generative AI have to do with reliability, in a way that I believe can never be solved, given their inherent nature.”
Our reading: Generative AI's reliability failings can never be solved, given its inherent nature.
No deadline stated Evidence as of
Limitation: The cited empirical study covers two benchmarks and a finite set of agents.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Causal claim
“Customers can easily switch to other providers, almost anytime they want, and for month there had been more and more alternatives, including Anthropic and Google, at more and more attractive prices, forcing OpenAI itself to cut prices just a few days ago.”
Our reading: Customers can easily switch LLM providers, and growing cheaper alternatives from Anthropic and Google forced OpenAI to cut prices in July 2024.
No horizon Evidence as of
Limitation: Company launch claims are not independent price-performance measurements.
Assessment: UnresolvedSaid
Observation
“LLMs have just became exactly the commodity I predicted they would become, at the lowest possible price.”
Our reading: LLMs have become a commodity available for free.
No horizon Evidence as of
Limitation: Capability comparisons in the cited primary sources come from Meta.
Assessment: UnresolvedSaid
Forecast
“Obviously, their only hope is to raise more cash, and they will certainly try.”
Our reading: OpenAI's only hope of survival is raising more cash, and it will try to.
No deadline stated Evidence as of
Limitation: This verdict retains both parts of the supplied claim; it does not silently reduce it to the fundraising prediction.
Assessment: UnresolvedSaid
Observation
“And leaving more and more code writing to generative AI, which grasps syntax but not meaning, as a new benchmark from Stephen Wolfram shows, is not the answer, either.”
Our reading: Generative AI grasps syntax but not meaning, as shown by a new Stephen Wolfram benchmark.
No horizon Evidence as of
Limitation: This is an evidence gap, not a finding that Marcus misreported the benchmark.
Source: Don’t look up: the massive Microsoft/Crowdstrike data outage is a huge wake-up call
Assessment: SupportedSaid
Causal claim
“As Ernie Davis and I pointed out in Rebooting AI, five years ago, part of the reason we are struggling with AI in complex AI systems is that we still lack adequate techniques for engineering complex systems. That hasn’t really changed.”
Our reading: Struggles with complex AI systems stem partly from a lack of adequate techniques for engineering complex systems, which has not really changed since 2019.
No horizon Evidence as of
Limitation: The causal assessment synthesizes technical explanations; it is not a controlled estimate of engineering's contribution to failures.
Source: Don’t look up: the massive Microsoft/Crowdstrike data outage is a huge wake-up call
Assessment: UnresolvedSaid
Forecast
“Neurosymbolic AI has long been an underdog; in the end I expect it to come from behind and be essential.”
Our reading: Neurosymbolic AI will eventually come from behind and become essential.
No deadline stated Evidence as of
Limitation: The intended destination and meaning of essential require interpretation.
Source: Marcus goes gaga over Gates clip
Assessment: SupportedSaid
Forecast
“As of today, I am more confident than ever that GPT-5 won’t land this year.”
Our reading: OpenAI will not release GPT-5 in 2024.
Deadline passed Evidence as of
Limitation: This record assesses only the release timing of a model named GPT-5. It does not assess Marcus's reasons for the forecast or what GPT-5 could do.
Assessment: UnresolvedSaid
Observation
“Others are bailing now, too. Or at least trying to subtly alter their positions, committing less to the unrealistic.”
Our reading: Prominent AI figures are now retreating from, or quietly softening, their unrealistic claims about LLMs and AGI.
No horizon Evidence as of
Limitation: Restricting others solely to the two named examples would produce a narrower assessment than the wording clearly warrants.
Assessment: SupportedSaid
Observation
“Exhibit A: OpenAI’s CTO Mira Murati just publicly acknowledged what I long suspected: there is no mind blowing GPT-5 behind the scenes as of yet.”
Our reading: As of June 2024, OpenAI has no mind-blowing GPT-5 model waiting behind the scenes.
No horizon Evidence as of
Limitation: The decisive source is contemporaneous news reporting because the relevant primary interview transcript could not be recovered.
Assessment: UnresolvedSaid
Causal claim
“LLMs are not AGI, and (on their own) never will be; scaling alone was never going to be enough.”
Our reading: Scaling alone is not enough to reach AGI.
No horizon Evidence as of
Limitation: Finite scaling experiments cannot establish what every larger system could achieve.
Assessment: SupportedSaid
Observation
“In many ways, it is much weaker than what Altman and I talked about at the Senate last year.”
Our reading: SB-1047 is in many ways much weaker than the AI regulation Marcus and Sam Altman discussed at the Senate in 2023.
No horizon Evidence as of
Limitation: The result depends on reading strength through the licensing and review mechanisms highlighted in the surrounding passage.
Assessment: UnresolvedSaid
Observation
“Anyone who can afford that can easily fill out some paperwork and run a few tests, just like I can pay a few bucks to get a driver’s license. It’s the least they can do.”
Our reading: Companies that can afford $100M+ training runs can easily afford SB-1047's paperwork and testing requirements.
No horizon Evidence as of
Limitation: The packet contains no documented estimate of the June proposal's incremental paperwork and testing costs matched to developers' available resources and existing commitments.
Assessment: ContradictedSaid
Observation
“He doesn’t specify which actual restrictions in the proposal bill would “stifle AI models”.”
Our reading: Andrew Ng's critique does not identify which specific SB-1047 provisions would stifle AI model builders.
No horizon Evidence as of
Limitation: This assessment does not validate Ng's legal interpretation or his prediction about innovation.
Assessment: UnresolvedSaid
Observation
“Every company that would be subject to the stronger set of regulations is valued in the billions.”
Our reading: Every company that would be subject to SB-1047's stronger regulations is valued in the billions of dollars.
No horizon Evidence as of
Limitation: Private-company valuations are intermittent and may refer to different share classes or transaction terms.
Assessment: UnresolvedSaid
Observation
“It can do some things better, some worse, very few reliably.”
Our reading: Compared with a smart high schooler, GPT-4 can do some things better and some things worse, and it can do very few things reliably.
No horizon Evidence as of
Limitation: Both sources are company disclosures from 2023 and do not establish the performance of an identified June 2024 GPT-4 version [e1][e2].
Source: AGI by 2027?
Assessment: UnresolvedSaid
Observation
“The double Y-axis makes no sense, and presupposes its own conclusion.”
Our reading: The double Y-axis on Aschenbrenner's graph makes no sense and presupposes its own conclusion.
No horizon Evidence as of
Limitation: The phrase makes no sense remains an evaluative judgment.
Source: AGI by 2027?
Assessment: MixedSaid
Observation
“The graph assumes that all continued progress will be same — ignoring qualitatively unsolved problems (hallucinations, planning, reasoning etc), the fact that synthetic data may not be as useful as natural data, limits on then-possible energy distribution and compute, etc.”
Our reading: Aschenbrenner's graph assumes all continued progress will be the same, ignoring qualitatively unsolved problems such as hallucinations, planning and reasoning, the possibility that synthetic data may not be as useful as natural data, and limits on energy distribution and compute.
No horizon Evidence as of
Limitation: The result depends on including the graph's supporting analysis.
Source: AGI by 2027?
Assessment: SupportedSaid
Observation
“For one thing, the bad press about Sam Altman and OpenAI, who once seemingly could do no wrong, just keeps coming.”
Our reading: Negative press coverage of Sam Altman and OpenAI, once seen as able to do no wrong, keeps coming.
No horizon Evidence as of
Limitation: This is a finding about the existence of repeated coverage, not its prevalence or rate of increase.
Source: AI as Reality Television
Assessment: UnresolvedSaid
Observation
“In 2019 I pointed out through a series of examples on Twitter with GPT-2 that LLMs were subject to essentially the same critique: greedy with respect to data, uninterpretable, unreliable as reasoners, subject to hallucinations, etc, that they lacked world models (“they don't develop robust representations of *how events unfold over time*), and that they would be inadequate on their own.”
Our reading: In 2019, using a series of GPT-2 examples on Twitter, Marcus pointed out that LLMs were greedy with respect to data, uninterpretable, unreliable as reasoners, and subject to hallucinations, that they lacked world models, and that they would be inadequate on their own.
No horizon Evidence as of
Limitation: The reply does not document the boundaries or full contents of the claimed 2019 GPT-2 Twitter series [e1].
Assessment: UnresolvedSaid
Observation
“Nowadays we both (unlike many other people) stress the absolutely essential nature of common sense, physical reasoning and world models, and the failure of current architectures to handle those well.”
Our reading: Marcus and LeCun, unlike many other people, now both stress that common sense, physical reasoning and world models are absolutely essential, and that current architectures fail to handle them well.
No horizon Evidence as of
Limitation: Public agreement does not establish who originated an idea or whether one speaker changed his position.
Assessment: UnresolvedSaid
Observation
“Worse, we are no closer to a solution to the alignment problem now than we were then.”
Our reading: There has been no progress toward solving the AI alignment problem between 2022 and mid-2024.
No horizon Evidence as of
Limitation: A narrower interpretation covering any improvement in present-model alignment would favor contradiction.
Assessment: SupportedSaid
Observation
“For all the daily claims of “exponential progress”, and for all the tens of billions that have been invested since, reliability is still a dream, and of course toxicity and hallucinations are still with us.”
Our reading: Despite tens of billions of dollars invested since 2022, deep learning systems are still unreliable and still produce toxicity and hallucinations.
No horizon Evidence as of
Limitation: The reliability phrase is rhetorical and supplies no threshold.
Assessment: UnresolvedSaid
Observation
“But OpenAI has been and continues to be a California-registered 501(3)(c) nonprofit, pledged not to profits, but to humanity – and still getting tax-exempt status on the basis of their promise to deploy AI safely:”
Our reading: OpenAI is a registered nonprofit receiving tax-exempt status based on its promise to deploy AI safely.
No horizon Evidence as of
Limitation: The packet lacks dated records confirming continued California registration and federal exemption through May 30, 2024; this does not establish that either had lapsed.
Source: Disturbing and misleading efforts to defame Helen Toner through misdirection
Assessment: UnresolvedSaid
Observation
“The 3.5 API (geared towards programmers as a backend tool) was available beforehand, but that’s not of course the consumer-directed product with gimmicks like word-by-word typing to give a false impression of humanness that ultimately went viral.”
Our reading: ChatGPT, unlike the earlier GPT-3.5 API, was a consumer product using gimmicks like word-by-word typing to give a false impression of humanness.
No horizon Evidence as of
Limitation: Earlier model availability rests on a contemporaneous community report, which does not identify an official product named GPT-3.5 API [e2].
Source: Disturbing and misleading efforts to defame Helen Toner through misdirection
Assessment: UnresolvedSaid
Observation
“Fundamentally, the essay attacks a strawman, viz. that the firing of Altman was somehow precipitated by an immediate safety concern), thereby diverting from the board’s actual, stated concern (candor).”
Our reading: The Taylor/Summers essay attacks a strawman about safety, diverting from the board's actual concern of candor in firing Altman.
No horizon Evidence as of
Limitation: The complete essay was unavailable; the retrieved reporting cannot establish what the full text omitted.
Source: Disturbing and misleading efforts to defame Helen Toner through misdirection
Assessment: UnresolvedSaid
Observation
“And oops, Grok’s basic understanding of time is shaky, too:”
Our reading: Grok has a shaky basic understanding of time.
No horizon Evidence as of
Limitation: A missing public test is not evidence that Grok performed well.
Assessment: UnresolvedSaid
Observation
“Musk is just building bigger and bigger LLMs, and Meta is mostly (despite LeCun’s reservations) mostly doing the same.”
Our reading: Musk's xAI is just building bigger LLMs, and Meta is mostly doing the same.
No horizon Evidence as of
Limitation: Public announcements do not provide a complete account of internal research.
Assessment: UnresolvedSaid
Causal claim
“By clustering piles of similar things together in a giant n-dimensional space, they pretty much automatically become synonym and pastische machines, regurgitating a lot of words with slight paraphrases while adding conceptually little, and understanding even less.”
Our reading: Because LLMs cluster similar things in a high-dimensional space, they become synonym and pastiche machines that regurgitate with slight paraphrase, adding little and understanding less.
No horizon Evidence as of
Limitation: Verbatim recall and memorization from similar sequences do not establish how often models produce close paraphrases in ordinary use [e1, e3].
Assessment: SupportedSaid
Observation
“Many people are reaching essential the same perceptions.”
Our reading: Many people are reaching essentially the same perceptions of Sam Altman.
No horizon Evidence as of
Limitation: The undefined word 'many' makes the result depend on a qualitative reading.
Source: Sam Altman’s fall from grace
Assessment: SupportedSaid
Observation
“In November, the narrative was about how everybody loved Sam. Things have changed. A lot.”
Our reading: The narrative about Sam Altman, which in November was that everybody loved him, has changed a lot.
No horizon Evidence as of
Limitation: The result depends on reading 'narrative' as prominent coverage and commentary, rather than measured population sentiment.
Source: Sam Altman’s fall from grace
Assessment: UnresolvedSaid
Observation
“It’s a pattern.”
Our reading: Sam Altman's lack of candor, seen in his statements to the Senate about OpenAI equity, his lies about Helen Toner and his lies about Scarlett Johansson, is a pattern.
No horizon Evidence as of
Limitation: The equity evidence does not establish whether the Senate answer was a misleading omission or material deception.
Source: The OpenAI board was right
Assessment: UnresolvedSaid
Observation
“I probably don’t have to tell you, but that’s complete bullshit. And stupid, obviously refuted bullshit at that.”
Our reading: OpenAI's claim that its voice's resemblance to Scarlett Johansson was coincidental is false.
No horizon Evidence as of
Limitation: Earlier casting evidence is context outside the denial's target period.
Source: The OpenAI board was right
Assessment: UnresolvedSaid
Observation
“They said it wasn’t intentional, but of course it was.”
Our reading: OpenAI intentionally made its voice resemble Scarlett Johansson's.
No horizon Evidence as of
Limitation: The primary production account comes from the company whose conduct is disputed.
Source: The OpenAI board was right
Assessment: UnresolvedSaid
Observation
“I can’t tell you what happened to Sam’s caps lock, but obviously the claim that the resemblance to ScarJo was a “coincidence” was a lie. Sam knew perfectly well what the character sounded like.”
Our reading: Sam Altman knew the voice resembled Johansson's Her character, so the coincidence claim was a lie.
No horizon Evidence as of
Limitation: The public record is largely circumstantial and includes interested-party accounts.
Source: The OpenAI board was right
Assessment: UnresolvedSaid
Observation
“Lots of quirky errors are already being reported, same as ever, like this reasoning error from Jane Rosenzweig:”
Our reading: Lots of quirky errors in GPT-4o are already being reported, same as ever, such as a reasoning error reported by Jane Rosenzweig.
No horizon Evidence as of
Limitation: Social-media text extraction omitted the underlying example.
Source: Hot take on OpenAI’s new GPT-4o
Assessment: UnresolvedSaid
Observation
“They don’t have GPT-5 after 14 months of trying.”
Our reading: OpenAI does not have GPT-5 after 14 months of trying since GPT-4.
No horizon Evidence as of
Limitation: Internal completion status may not be public.
Source: Hot take on OpenAI’s new GPT-4o
Assessment: UnresolvedSaid
Causal claim
“LLMs confined to X tweets will be even less grounded in reality.”
Our reading: LLMs whose synthesis is confined to X tweets will be even less grounded in reality than LLMs in general.
No horizon Evidence as of
Limitation: This is an unresolved causal assessment, not a finding that Marcus was wrong.
Assessment: SupportedSaid
Causal claim
“The very process that LLMs use to generalize also creates hallucinations.”
Our reading: The same process LLMs use to generalize is what creates their hallucinations.
No horizon Evidence as of
Limitation: The theory covers rare, arbitrary facts most clearly. Other kinds of hallucination may have other causes.
Assessment: MixedSaid
Observation
“Instead, everything that LLMs say -- true or false -- comes from the same process of statistically reconstructing what words are likely in some context. They NEVER fact-check what they say.”
Our reading: All LLM output, true or false, comes from the same statistical word-reconstruction process, and LLMs never fact-check what they say.
No horizon Evidence as of
Limitation: The packet documents next-token generation only for GPT-4 and does not cover every LLM in use during the period.
Assessment: UnresolvedSaid
Causal claim
“That means, of course, that OpenAI’s business model is threatened; same for Anthropic’s, Mistral’s, and so on.”
Our reading: Meta's free near-GPT-4 model threatens the business models of OpenAI, Anthropic, Mistral and similar companies.
No horizon Evidence as of
Limitation: Commercial threat is broader than benchmark performance.
Source: Groupthink versus Tulips. What will historians think of 2024?
Assessment: UnresolvedSaid
Forecast
“Whatever tiny moat there was between the companies will further diminish.”
Our reading: The small moat between LLM companies will further diminish.
No deadline stated Evidence as of
Limitation: The ordinary business meaning of moat is broader than model quality.
Source: Groupthink versus Tulips. What will historians think of 2024?
Assessment: SupportedSaid
Forecast
“LLMs are well on their way to becoming a commodity; price wars will be intense.”
Our reading: LLMs are becoming a commodity and price wars among providers will be intense.
Settled before its deadline Evidence as of
Limitation: Commoditization is assessed as a direction of travel, consistent with well on their way.
Source: Groupthink versus Tulips. What will historians think of 2024?
Assessment: UnresolvedSaid
Conditional forecast
“A billion robots with a superficial understanding of the physical world would be a recipe for disaster.”
Our reading: Deploying a billion robots with only superficial understanding of the physical world would lead to disaster.
No deadline stated Evidence as of
Only if: if robots have only superficial understanding of the physical world
Assessment: UnresolvedSaid
Forecast
“Khosla is a venture capitalist, and I think venture capitalists will have a huge role in whether or not we get to the promised land.”
Our reading: Marcus thinks venture capitalists will have a huge role in whether or not we get to the promised land of world-improving technology.
No deadline stated Evidence as of
Limitation: The destination and magnitude of influence remain qualitative.
Assessment: UnresolvedSaid
Observation
“Microsoft’s Mustafa Suleyman promised that hallucinations would be cured “soon”, yet my X feed is still filled with examples like these from Princeton Professor Aleksandra Korolova:”
Our reading: Although Mustafa Suleyman promised that hallucinations would be cured "soon", Marcus's X feed was still filled with examples of AI hallucinations, such as those from Princeton Professor Aleksandra Korolova.
No horizon Evidence as of
Limitation: Partial support is not a mixed outcome because no contradictory outcome evidence was found.
Assessment: UnresolvedSaid
Conditional forecast
“We won’t get to a billion personal robots if they are as dodgy as driverless cars, frequently working, yet stymied often enough by outliers that we can’t fully count on them.”
Our reading: A billion personal robots will not be deployed if robots are as unreliable with outliers as driverless cars are.
No deadline stated Evidence as of
Only if: if robots are as unreliable as driverless cars
Assessment: SupportedSaid
Observation
“Adding visual data to LLMs hasn’t solved hallucinations.”
Our reading: Adding visual data to LLMs has not solved hallucinations.
No horizon Evidence as of
Limitation: 'Solved' is not defined. The claim is read as hallucinations remaining a material problem, not as adding vision bringing no improvement at all. The evidence packet notes that the GPT-4V system card reports fewer hallucinations during a beta, but that passage is not in the cited excerpt.
Source: TED, AI, and the test of time
Assessment: UnresolvedSaid
Observation
“I think it’s fair to say that the multimodal stuff still is very buggy, having troubles with basics like the English alphabet.”
Our reading: OpenAI's multimodal capabilities remain very buggy as of April 2024, struggling even with basics like the English alphabet.
No horizon Evidence as of
Limitation: The assessment is limited by source-access failures.
Source: TED, AI, and the test of time
Assessment: UnresolvedSaid
Observation
“I warned that LLMs would be used by bad actors, and they increasingly are.”
Our reading: LLMs are increasingly being used by bad actors.
No horizon Evidence as of
Limitation: Malicious use can remain undetected.
Source: TED, AI, and the test of time
Assessment: UnresolvedSaid
Observation
“I am just not seeing that as OpenAI’s mission, anymore, especially when they have continued to fail to compensate most of the creators whose work they are drawing on, largely without consent.”
Our reading: OpenAI no longer appears to pursue its mission of ensuring AGI benefits all of humanity, and it has continued not to compensate most creators whose work it uses, largely without their consent.
No horizon Evidence as of
Limitation: The factual component is assessable in principle, but the required population and coverage records are missing.
Source: TED, AI, and the test of time
Assessment: MixedSaid
Observation
“The study Mollick linked to doesn’t actually show what he claims. If you read it carefully, it says literally nothing about capabilities improving.”
Our reading: The study Mollick cited does not show that LLM capabilities are improving; it only addresses compute efficiency.
No horizon Evidence as of
Limitation: Language-model performance is narrower than general capability.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Observation
“What it actually shows is some improvements, mostly modest, over the last few months for a bunch of different measures.”
Our reading: OpenAI's GPT-4 Turbo graph shows only mostly modest improvements over recent months.
No horizon Evidence as of
Limitation: Modest is a qualitative judgment, and relative changes can look larger.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: UnresolvedSaid
Observation
“Another way to think about this (see third graph above) is that there about 5 - 7 recent models are on a par with GPT-4, but none is clearly and decisively ahead.”
Our reading: About 5 to 7 recent models are on par with GPT-4, but none is clearly ahead.
No horizon Evidence as of
Limitation: The earlier four-model list does not contradict a later approximate five-to-seven-model count.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Observation
“I think it’s fair to say that they have not. GPT-Turbo released this week still produces its share of groaners, like this interchange Phil Libin just sent:”
Our reading: Hallucinations and stupid errors in LLMs have not been solved; GPT-4 Turbo still makes them.
No horizon Evidence as of
Limitation: The coding test measures task completion, not hallucination frequency.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: UnresolvedSaid
Observation
“One more way to look at this, is this graph I just saw: enormous convergence on GPT-4 level performance, in multiple models released since, yet nothing decisively ahead.”
Our reading: Multiple models released since GPT-4 have converged on GPT-4-level performance with none decisively ahead.
No horizon Evidence as of
Limitation: A coding benchmark cannot establish general capability parity.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: UnresolvedSaid
Conditional forecast
“If enthusiasm for GenAI dwindles and market valuations plummet, AI won’t disappear, and LLMs won’t disappear; they will still have their place as tools for statistical approximation.”
Our reading: If enthusiasm for GenAI dwindles and market valuations plummet, AI and LLMs will not disappear, and LLMs will still have their place as tools for statistical approximation.
No deadline stated Evidence as of
Only if: if enthusiasm for GenAI dwindles and market valuations plummet
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Observation
“Already in recent weeks, Inflection AI largely closed shop, Stability AI is struggling, and LLM-based autonomous vehicle company called Ghost closed shop, and a software engineer on YouTube raised quite serious questions about the wildly-hyped AI coding system Devin.”
Our reading: In recent weeks Inflection AI largely closed, Stability AI is struggling, Ghost closed, and serious questions were raised about Devin.
No horizon Evidence as of
Limitation: The Inflection finding depends on the meaning of 'largely'; the company continued operating.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Observation
“Huge jump from GPT-2 to GPT-3. Huge jump from GPT-3 to GPT-4 … and not so much for GPT-4 (13 months ago) to GPT-4 Turbo (just released). It’s hard not to see this plot as tentative evidence for the hypothesis of diminishing returns.”
Our reading: MMLU gains from GPT-4 to GPT-4 Turbo are small compared with earlier generational jumps, tentatively evidencing diminishing returns.
No horizon Evidence as of
Limitation: The records do not compare all four models under one protocol: GPT-2 was fine-tuned, GPT-3 used few-shot prompting, GPT-4 used five-shot prompting, and Turbo used zero-shot chain-of-thought prompting [e1, e2, e3].
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: UnresolvedSaid
Forecast
“I couldn’t agree more. And parallel arguments could be made in many domains.”
Our reading: Marcus endorses the view that ChatGPT and similar tools will not themselves be virtual scientists anytime soon, and says parallel arguments could be made in many domains.
No deadline stated Evidence as of
Limitation: “Anytime soon” supplies no deadline; reaching the evaluation date cannot establish support.
Source: Superhuman AGI is not nigh
Assessment: SupportedSaid
Observation
“Tesla FSD is getting better. But is getting enough better?”
Our reading: Tesla FSD is getting better, and he raises the question of whether it is getting enough better.
No horizon Evidence as of
Limitation: The tracker figures in e1 come from crowdsourced, self-reported data. e1 notes that data for later v12.3.x releases still needed weeks to be entered.
Source: 24 hours in AI news
Assessment: SupportedSaid
Observation
“current models aren’t literally look up tables; they can generalize to some degree, but not enough.”
Our reading: Current AI models can generalize to some degree but not enough.
No horizon Evidence as of
Limitation: The outcome depends on interpreting 'not enough' through the surrounding discussion of reliability.
Assessment: ContradictedSaid
Causal claim
“This is why driverless cars are still just demos, and why LLMs will never be reliable.”
Our reading: Poor generalization beyond training data is why driverless cars are still just demos.
No horizon Evidence as of
Limitation: The decisive operational account is the company's regulatory filing.
Assessment: UnresolvedSaid
Causal claim
“It also drives their greediness with data, and even the ethical choices their developers have been making.”
Our reading: Neural networks' trouble generalizing beyond their training space, at least in the configurations dominant over the last three decades, also drives their greediness with data and even the ethical choices their developers have been making.
No horizon Evidence as of
Limitation: The technical evidence concerns particular recurrent networks and tasks, and its link to data demand is tentative [e1].
Assessment: SupportedSaid
Observation
“Neural networks (at least in the configurations that have been dominant over the last three decades) have trouble generalizing beyond the multidimensional space that surrounds their training examples.”
Our reading: Neural networks in the configurations dominant for the last three decades have trouble generalizing beyond the space surrounding their training examples.
No horizon Evidence as of
Limitation: The outcome depends on reading the spatial language as unfamiliar conditions rather than a precise geometric boundary.
Assessment: UnresolvedSaid
Forecast
“In short, diminishing returns are what we can expect.”
Our reading: Scaling data for current AI models will yield diminishing returns.
No deadline stated Evidence as of
Limitation: A narrower claim about a specified loss curve could be supported by individual experiments.
Assessment: UnresolvedSaid
Forecast
“Sooner or later, the exponential greed for data will exceed what is available.”
Our reading: The exponentially growing demand for training data will eventually exceed the data available.
No deadline stated Evidence as of
Limitation: Epoch's quantitative estimate covers public human-generated text, not all usable training data [e1].
Assessment: UnresolvedSaid
Observation
“an intellectual monoculture — with no alternative approach that is nearly as well-funded.”
Our reading: The AI field is an intellectual monoculture in which no alternative approach is nearly as well-funded as generative AI.
No horizon Evidence as of
Limitation: The annual reporting period is inferred.
Assessment: SupportedSaid
Observation
“The good news is that word is finally getting out more broadly.”
Our reading: Awareness of the danger generative AI poses to the information ecosystem is finally spreading more widely.
No horizon Evidence as of
Limitation: The trend ('more broadly') is inferred from high levels, not from a measured before-and-after comparison.
Assessment: SupportedSaid
Observation
“Meanwhile X itself is promoting nonsense.”
Our reading: X itself is promoting nonsense.
No horizon Evidence as of
Limitation: The assessment relies on direct news reporting with screenshots and an independent fact-check; no separately retrievable original X display was obtained.
Assessment: SupportedSaid
Causal claim
“Ironic that his own chatbot, Grok, is making the problem worse.”
Our reading: Musk's own chatbot, Grok, is making the information-pollution problem worse.
No horizon Evidence as of
Limitation: The false Iran story began with user posts; Grok's added effect on spread is not separately measured.
Assessment: SupportedSaid
Observation
“The bad news, which outweighs the good, is that the problem is quickly getting worse.”
Our reading: Generative-AI pollution of the information ecosystem is getting worse quickly as of April 2024.
No horizon Evidence as of
Limitation: Tracker counts come from one commercial source and may partly reflect more searching rather than only more sites.
Assessment: UnresolvedSaid
Observation
“At the same time, more optimistic predictions like these [from Reid Hoffman September 7, 2023] have not thus far borne out.”
Our reading: Reid Hoffman's optimistic September 2023 AI predictions had not come true as of March 2024.
No horizon Evidence as of
Limitation: The interview identifies the hallucination prediction but does not establish the complete set of predictions Marcus referenced. [e4]
Assessment: SupportedSaid
Observation
“The trouble is that the bigger the model, the higher the internal costs. And current-sized models, already expensive, aren’t reliable enough for many jobs.”
Our reading: Bigger models have higher costs, and current-sized models are already expensive and not reliable enough for many jobs.
No horizon Evidence as of
Limitation: The result depends on comparing otherwise similar implementations and interpreting reliability as suitability for unverified use.
Assessment: UnresolvedSaid
Observation
“So far most uses of LLMs have been for testing, not production-level systems.”
Our reading: Most uses of LLMs so far have been for testing rather than production systems.
No horizon Evidence as of
Limitation: Survey respondents are not interchangeable with projects or usage volume.
Assessment: UnresolvedSaid
Causal claim
“They are inherent to how the system makes predictions, and to the fact that LLMs themselves don’t have the wherewithal to look things up and reason over them.”
Our reading: Hallucinations are inherent to how LLMs make predictions and to their inability to look things up and reason over them.
No horizon Evidence as of
Limitation: A conditional mathematical result is not a universal result about deployed systems.
Assessment: UnresolvedSaid
Causal claim
“Meanwhile hype itself is becoming a kind of risk factor. The more hype, the bigger the fall, if expectations aren’t met.”
Our reading: Hype around generative AI is a risk factor: the more hype, the bigger the fall if expectations are not met.
No horizon Evidence as of
Limitation: The closest paper covers AI broadly and a particular national market.
Assessment: UnresolvedSaid
Observation
“Thus far, the closest thing to a killer app is probably coding assistace, but the income there isn’t massive enough to cover costs of the chips, legal etc.”
Our reading: Coding assistance is probably the closest thing to a generative AI killer app, but its income is not large enough to cover chip, legal and other costs.
No horizon Evidence as of
Limitation: The available financial reporting is secondary and concerns one product.
Assessment: UnresolvedSaid
Forecast
“Core problems like hallucinations aren’t going way.”
Our reading: Core generative AI problems like hallucinations are not going away.
No deadline stated Evidence as of
Limitation: The cited persistence observation is from September 2025, not a complete audit of every system available at the evaluation date.
Assessment: UnresolvedSaid
Causal claim
“(Not easy. With everyone building similar models, there is no moat, hence lots of competition on price).”
Our reading: Because everyone is building similar models, generative AI companies have no moat, leading to heavy price competition.
No horizon Evidence as of
Limitation: Observed convergence does not establish universal substitutability.
Assessment: UnresolvedSaid
Observation
“$50B in, $3B out. That’s obviously not sustainable.”
Our reading: Generative AI investment of about $50B against $3B in revenue is obviously not sustainable.
No horizon Evidence as of
Limitation: The repeated estimate is not an independent validation of the amounts, their coverage or whether revenue was realized or annualized. [e1]
Assessment: UnresolvedSaid
Observation
“Most of those who invested in driverless cars lost money, unless they invested early and cashed out, before patience began to run out.”
Our reading: Most of those who invested in driverless cars lost money, unless they invested early and cashed out before patience began to run out.
No horizon Evidence as of
Limitation: The evidence covers selected investments, not a representative population.
Source: The Second Worst $100B Investment in The History of AI?
Assessment: UnresolvedSaid
Observation
“There may be a payoff someday, but thus far there has not been a lot.”
Our reading: As of March 2024, investment in driverless cars has not produced much payoff.
No horizon Evidence as of
Limitation: The company examples are not a representative industry return calculation.
Source: The Second Worst $100B Investment in The History of AI?
Assessment: SupportedSaid
Observation
“As noted here before, one study indicates that code quality is going down.”
Our reading: One study indicates code quality is going down amid AI-assisted coding.
No horizon Evidence as of
Limitation: The study uses indirect quality indicators.
Source: The race between positive and negative applications of Generative AI is on – and not looking pretty
Assessment: SupportedSaid
Observation
“On the other hand, the use cases on the negative side of the ledger are proliferating fast, literally faster than I can report them here, from fake books (undermining authors) to rings of fake websites designed to sell ads”
Our reading: Harmful uses of generative AI (fake books, ad-farm websites, election misinformation, polluted science) are proliferating rapidly.
No horizon Evidence as of
Limitation: Support depends on reading proliferation as broad expansion rather than proven rapid growth in every category.
Source: The race between positive and negative applications of Generative AI is on – and not looking pretty
Assessment: UnresolvedSaid
Observation
“(OpenAI even brazenly asked the British goverment to exempt them from copyright laws, wanting all the gain at none of the cost.)”
Our reading: OpenAI asked the British government to exempt it from copyright laws.
No horizon Evidence as of
Limitation: The official submission required an accessible mirror.
Source: The race between positive and negative applications of Generative AI is on – and not looking pretty
Assessment: SupportedSaid
Observation
“Some core problems have lingered, particularly hallucinations, instability and occasional nonsensical errors.”
Our reading: Hallucinations, instability and occasional nonsensical errors remain unsolved in GPT-4 class LLMs.
No horizon Evidence as of
Limitation: The research tests selected tasks rather than representative everyday usage.
Source: Happy 1st Birthday, GPT-4
Assessment: SupportedSaid
Forecast
“We can expect price wars.”
Our reading: There will be price wars among LLM providers.
Settled before its deadline Evidence as of
Limitation: The decisive episode occurred in China; the claim supplied no geographic restriction.
Source: Happy 1st Birthday, GPT-4
Assessment: UnresolvedSaid
Observation
“Translation: But I do have equity in YCombinator and YCombinator has equity in OpenAI. It’s probably worth a lot, but you wouldn’t like me as much if I acknowledged that, so I won’t. Also, did I mention I own the OpenAI venture fund? Nope, guess not, sorry.”
Our reading: In his May 2023 Senate testimony, Sam Altman left out his indirect OpenAI stake through Y Combinator and his ownership of the OpenAI venture fund.
No horizon Evidence as of
Limitation: This is an evidentiary hold, not a finding that either interest was absent.
Source: OpenAI’s Lies and Half-Truths
Assessment: SupportedSaid
Observation
“Translation: But (revealed a few weeks later) behind the scenes we will try hard to water down your regulations.”
Our reading: While publicly asking the Senate to regulate it, OpenAI worked behind the scenes to water down regulations.
No horizon Evidence as of
Limitation: The documented request concerned the EU AI Act; it does not establish efforts to weaken a U.S. Senate proposal [e2].
Source: OpenAI’s Lies and Half-Truths
Assessment: UnresolvedSaid
Observation
“Two clearest examples so far this year, are the now retracted paper with the above-mentioned rat and a new lithium battery paper, with an opening sentence clearly written by an LLM.”
Our reading: In 2024 so far, a now-retracted paper with an AI-generated rat image and a lithium battery paper with an LLM-written opening sentence are the clearest examples of GenAI contaminating science.
No horizon Evidence as of
Limitation: The factual examples are strongly corroborated, but only the rat example was verified within the target period.
Source: The Year of the Big-Balled Rat
Assessment: SupportedSaid
Observation
“GenAI is messing with science.”
Our reading: Generative AI is currently damaging science.
No horizon Evidence as of
Limitation: The conclusion concerns an identifiable publication failure, not the frequency or aggregate consequences of such failures.
Source: The Year of the Big-Balled Rat
Assessment: SupportedSaid
Observation
“And the apparent copyright infringements that Reid Southen and I discussed in early January continue apace two months later, as I found in a quick experiment this morning:”
Our reading: The apparent copyright infringements that Marcus and Reid Southen discussed in early January 2024 were continuing apace two months later, based on a quick experiment Marcus ran on the morning of the post.
No horizon Evidence as of
Limitation: The result depends on reading 'apace' as continued ready production rather than a quantitative rate claim.
Source: OpenAI’s “Own Goal”
Assessment: SupportedSaid
Observation
“Almost nothing is known about the precise internal mechanisms of GPT-4 or what it has trained on.”
Our reading: Almost nothing is known about the precise internal mechanisms of GPT-4 or what it has been trained on.
No horizon Evidence as of
Limitation: Official nondisclosure does not prove that outsiders knew nothing.
Source: OpenAI’s “Own Goal”
Assessment: SupportedSaid
Observation
“They knew they were misleading people and failed to take an opportunity to correct.”
Our reading: OpenAI knew it was misleading people into believing it meant open-sourced AI and failed to take an opportunity, at a May 2016 meeting, to correct that belief.
No horizon Evidence as of
Limitation: The supplied evidence consists of one reported account; no independent recording, transcript, or meeting minutes are supplied.
Source: OpenAI’s “Own Goal”
Assessment: SupportedSaid
Observation
“On overt measures, the systems were fine. On covert measures, they were a disaster:”
Our reading: LLMs tested looked fine on overt racism measures but did very badly on covert racism measures when prompted with African American English.
No horizon Evidence as of
Limitation: The abstract does not establish that every tested model was free of overt bias.
Source: Covert racism in LLMs
Assessment: UnresolvedSaid
Causal claim
“Musk's lawsuit, combined with OpenAI's bizarre arrangement with Microsoft in which the continuation of an exclusive license rests on whether or not OpenAI has achieved something that is difficult to define (AGI) has left OpenAI in the position of needing skeptical experts to deflate their own hype.”
Our reading: Musk's lawsuit plus the AGI-contingent Microsoft license have put OpenAI in the position of needing skeptical experts to deflate its own hype.
No horizon Evidence as of
Limitation: A practical incentive is easier to establish than necessity.
Source: The Irony and the Agony of Elon Musk's Lawsuit against OpenAI
Assessment: SupportedSaid
Observation
“Dropped is any mention of (a) the nonprofit (even though legally the company remains a nonprofit), (b) open source, once central both to their mission and name, and (c) the original phrasing about being "unconstrained by a need to generate financial return."”
Our reading: OpenAI's current homepage mission statement no longer mentions the nonprofit, open source, or being unconstrained by a need to generate financial return.
No horizon Evidence as of
Limitation: The assessment covers text preserved and extracted from the archive, not unextracted video or image content.
Source: The Irony and the Agony of Elon Musk's Lawsuit against OpenAI
Assessment: MixedSaid
Observation
“The company Sam and Greg built has little to do with what was originally promised.”
Our reading: The OpenAI that Altman and Brockman built has little to do with what was originally promised.
No horizon Evidence as of
Limitation: The weighting of different founding commitments is an evaluator interpretation.
Source: Elon Musk sues OpenAI