Claims checked against evidence, page 10 of 15
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Claims checked against evidence (1432)
Each of these claims was checked against dated evidence.
Page 10 of 15: claims 901 to 1000 of 1432, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Gary Marcus Causal claim
“Ironic that his own chatbot, Grok, is making the problem worse.”
Our reading: Musk's own chatbot, Grok, is making the information-pollution problem worse.
No horizon Evidence as of
Limitation: The false Iran story began with user posts; Grok's added effect on spread is not separately measured.
Assessment: SupportedSaid
Gary Marcus Observation
“The bad news, which outweighs the good, is that the problem is quickly getting worse.”
Our reading: Generative-AI pollution of the information ecosystem is getting worse quickly as of April 2024.
No horizon Evidence as of
Limitation: Tracker counts come from one commercial source and may partly reflect more searching rather than only more sites.
Assessment: UnresolvedSaid
Gary Marcus Observation
“At the same time, more optimistic predictions like these [from Reid Hoffman September 7, 2023] have not thus far borne out.”
Our reading: Reid Hoffman's optimistic September 2023 AI predictions had not come true as of March 2024.
No horizon Evidence as of
Limitation: The interview identifies the hallucination prediction but does not establish the complete set of predictions Marcus referenced. [e4]
Assessment: SupportedSaid
Gary Marcus Observation
“The trouble is that the bigger the model, the higher the internal costs. And current-sized models, already expensive, aren’t reliable enough for many jobs.”
Our reading: Bigger models have higher costs, and current-sized models are already expensive and not reliable enough for many jobs.
No horizon Evidence as of
Limitation: The result depends on comparing otherwise similar implementations and interpreting reliability as suitability for unverified use.
Assessment: UnresolvedSaid
Gary Marcus Observation
“So far most uses of LLMs have been for testing, not production-level systems.”
Our reading: Most uses of LLMs so far have been for testing rather than production systems.
No horizon Evidence as of
Limitation: Survey respondents are not interchangeable with projects or usage volume.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“They are inherent to how the system makes predictions, and to the fact that LLMs themselves don’t have the wherewithal to look things up and reason over them.”
Our reading: Hallucinations are inherent to how LLMs make predictions and to their inability to look things up and reason over them.
No horizon Evidence as of
Limitation: A conditional mathematical result is not a universal result about deployed systems.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Meanwhile hype itself is becoming a kind of risk factor. The more hype, the bigger the fall, if expectations aren’t met.”
Our reading: Hype around generative AI is a risk factor: the more hype, the bigger the fall if expectations are not met.
No horizon Evidence as of
Limitation: The closest paper covers AI broadly and a particular national market.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Thus far, the closest thing to a killer app is probably coding assistace, but the income there isn’t massive enough to cover costs of the chips, legal etc.”
Our reading: Coding assistance is probably the closest thing to a generative AI killer app, but its income is not large enough to cover chip, legal and other costs.
No horizon Evidence as of
Limitation: The available financial reporting is secondary and concerns one product.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Core problems like hallucinations aren’t going way.”
Our reading: Core generative AI problems like hallucinations are not going away.
No deadline stated Evidence as of
Limitation: The cited persistence observation is from September 2025, not a complete audit of every system available at the evaluation date.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“(Not easy. With everyone building similar models, there is no moat, hence lots of competition on price).”
Our reading: Because everyone is building similar models, generative AI companies have no moat, leading to heavy price competition.
No horizon Evidence as of
Limitation: Observed convergence does not establish universal substitutability.
Assessment: UnresolvedSaid
Gary Marcus Observation
“$50B in, $3B out. That’s obviously not sustainable.”
Our reading: Generative AI investment of about $50B against $3B in revenue is obviously not sustainable.
No horizon Evidence as of
Limitation: The repeated estimate is not an independent validation of the amounts, their coverage or whether revenue was realized or annualized. [e1]
Assessment: UnresolvedSaid
Gary Marcus Observation
“Most of those who invested in driverless cars lost money, unless they invested early and cashed out, before patience began to run out.”
Our reading: Most of those who invested in driverless cars lost money, unless they invested early and cashed out before patience began to run out.
No horizon Evidence as of
Limitation: The evidence covers selected investments, not a representative population.
Source: The Second Worst $100B Investment in The History of AI?
Assessment: UnresolvedSaid
Gary Marcus Observation
“There may be a payoff someday, but thus far there has not been a lot.”
Our reading: As of March 2024, investment in driverless cars has not produced much payoff.
No horizon Evidence as of
Limitation: The company examples are not a representative industry return calculation.
Source: The Second Worst $100B Investment in The History of AI?
Assessment: SupportedSaid
Gary Marcus Observation
“As noted here before, one study indicates that code quality is going down.”
Our reading: One study indicates code quality is going down amid AI-assisted coding.
No horizon Evidence as of
Limitation: The study uses indirect quality indicators.
Source: The race between positive and negative applications of Generative AI is on – and not looking pretty
Assessment: SupportedSaid
Gary Marcus Observation
“On the other hand, the use cases on the negative side of the ledger are proliferating fast, literally faster than I can report them here, from fake books (undermining authors) to rings of fake websites designed to sell ads”
Our reading: Harmful uses of generative AI (fake books, ad-farm websites, election misinformation, polluted science) are proliferating rapidly.
No horizon Evidence as of
Limitation: Support depends on reading proliferation as broad expansion rather than proven rapid growth in every category.
Source: The race between positive and negative applications of Generative AI is on – and not looking pretty
Assessment: UnresolvedSaid
Gary Marcus Observation
“(OpenAI even brazenly asked the British goverment to exempt them from copyright laws, wanting all the gain at none of the cost.)”
Our reading: OpenAI asked the British government to exempt it from copyright laws.
No horizon Evidence as of
Limitation: The official submission required an accessible mirror.
Source: The race between positive and negative applications of Generative AI is on – and not looking pretty
Assessment: SupportedSaid
Gary Marcus Observation
“Some core problems have lingered, particularly hallucinations, instability and occasional nonsensical errors.”
Our reading: Hallucinations, instability and occasional nonsensical errors remain unsolved in GPT-4 class LLMs.
No horizon Evidence as of
Limitation: The research tests selected tasks rather than representative everyday usage.
Source: Happy 1st Birthday, GPT-4
Assessment: SupportedSaid
Gary Marcus Forecast
“We can expect price wars.”
Our reading: There will be price wars among LLM providers.
Settled before its deadline Evidence as of
Limitation: The decisive episode occurred in China; the claim supplied no geographic restriction.
Source: Happy 1st Birthday, GPT-4
Assessment: UnresolvedSaid
Gary Marcus Observation
“Translation: But I do have equity in YCombinator and YCombinator has equity in OpenAI. It’s probably worth a lot, but you wouldn’t like me as much if I acknowledged that, so I won’t. Also, did I mention I own the OpenAI venture fund? Nope, guess not, sorry.”
Our reading: In his May 2023 Senate testimony, Sam Altman left out his indirect OpenAI stake through Y Combinator and his ownership of the OpenAI venture fund.
No horizon Evidence as of
Limitation: This is an evidentiary hold, not a finding that either interest was absent.
Source: OpenAI’s Lies and Half-Truths
Assessment: SupportedSaid
Gary Marcus Observation
“Translation: But (revealed a few weeks later) behind the scenes we will try hard to water down your regulations.”
Our reading: While publicly asking the Senate to regulate it, OpenAI worked behind the scenes to water down regulations.
No horizon Evidence as of
Limitation: The documented request concerned the EU AI Act; it does not establish efforts to weaken a U.S. Senate proposal [e2].
Source: OpenAI’s Lies and Half-Truths
Assessment: UnresolvedSaid
Gary Marcus Observation
“Two clearest examples so far this year, are the now retracted paper with the above-mentioned rat and a new lithium battery paper, with an opening sentence clearly written by an LLM.”
Our reading: In 2024 so far, a now-retracted paper with an AI-generated rat image and a lithium battery paper with an LLM-written opening sentence are the clearest examples of GenAI contaminating science.
No horizon Evidence as of
Limitation: The factual examples are strongly corroborated, but only the rat example was verified within the target period.
Source: The Year of the Big-Balled Rat
Assessment: SupportedSaid
Gary Marcus Observation
“GenAI is messing with science.”
Our reading: Generative AI is currently damaging science.
No horizon Evidence as of
Limitation: The conclusion concerns an identifiable publication failure, not the frequency or aggregate consequences of such failures.
Source: The Year of the Big-Balled Rat
Assessment: SupportedSaid
Gary Marcus Observation
“And the apparent copyright infringements that Reid Southen and I discussed in early January continue apace two months later, as I found in a quick experiment this morning:”
Our reading: The apparent copyright infringements that Marcus and Reid Southen discussed in early January 2024 were continuing apace two months later, based on a quick experiment Marcus ran on the morning of the post.
No horizon Evidence as of
Limitation: The result depends on reading 'apace' as continued ready production rather than a quantitative rate claim.
Source: OpenAI’s “Own Goal”
Assessment: SupportedSaid
Gary Marcus Observation
“Almost nothing is known about the precise internal mechanisms of GPT-4 or what it has trained on.”
Our reading: Almost nothing is known about the precise internal mechanisms of GPT-4 or what it has been trained on.
No horizon Evidence as of
Limitation: Official nondisclosure does not prove that outsiders knew nothing.
Source: OpenAI’s “Own Goal”
Assessment: SupportedSaid
Gary Marcus Observation
“They knew they were misleading people and failed to take an opportunity to correct.”
Our reading: OpenAI knew it was misleading people into believing it meant open-sourced AI and failed to take an opportunity, at a May 2016 meeting, to correct that belief.
No horizon Evidence as of
Limitation: The supplied evidence consists of one reported account; no independent recording, transcript, or meeting minutes are supplied.
Source: OpenAI’s “Own Goal”
Assessment: SupportedSaid
Gary Marcus Observation
“On overt measures, the systems were fine. On covert measures, they were a disaster:”
Our reading: LLMs tested looked fine on overt racism measures but did very badly on covert racism measures when prompted with African American English.
No horizon Evidence as of
Limitation: The abstract does not establish that every tested model was free of overt bias.
Source: Covert racism in LLMs
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Musk's lawsuit, combined with OpenAI's bizarre arrangement with Microsoft in which the continuation of an exclusive license rests on whether or not OpenAI has achieved something that is difficult to define (AGI) has left OpenAI in the position of needing skeptical experts to deflate their own hype.”
Our reading: Musk's lawsuit plus the AGI-contingent Microsoft license have put OpenAI in the position of needing skeptical experts to deflate its own hype.
No horizon Evidence as of
Limitation: A practical incentive is easier to establish than necessity.
Source: The Irony and the Agony of Elon Musk's Lawsuit against OpenAI
Assessment: SupportedSaid
Gary Marcus Observation
“Dropped is any mention of (a) the nonprofit (even though legally the company remains a nonprofit), (b) open source, once central both to their mission and name, and (c) the original phrasing about being "unconstrained by a need to generate financial return."”
Our reading: OpenAI's current homepage mission statement no longer mentions the nonprofit, open source, or being unconstrained by a need to generate financial return.
No horizon Evidence as of
Limitation: The assessment covers text preserved and extracted from the archive, not unextracted video or image content.
Source: The Irony and the Agony of Elon Musk's Lawsuit against OpenAI
Assessment: MixedSaid
Gary Marcus Observation
“The company Sam and Greg built has little to do with what was originally promised.”
Our reading: The OpenAI that Altman and Brockman built has little to do with what was originally promised.
No horizon Evidence as of
Limitation: The weighting of different founding commitments is an evaluator interpretation.
Source: Elon Musk sues OpenAI
Assessment: UnresolvedSaid
Gary Marcus Observation
“I take the recital of facts in Musk’s lawsuit to be largely accurate. It is consistent with my own observations and with earlier reporting by The New York Times and The New Yorker.”
Our reading: The facts set out in Musk's lawsuit against OpenAI are largely accurate.
No horizon Evidence as of
Limitation: The original state complaint and later federal pleadings are different documents.
Source: Elon Musk sues OpenAI
Assessment: UnresolvedSaid
Gary Marcus Observation
“Elon did not get what he paid and worked for.”
Our reading: Elon Musk did not get the OpenAI he paid for and worked for.
No horizon Evidence as of
Limitation: The record does not establish Musk's acceptance of the specific structure ultimately adopted [e3].
Source: Elon Musk sues OpenAI
Assessment: SupportedSaid
Gary Marcus Observation
“That’s exactly what classical AI has been trying to do for decades, with only limited success. And what systems like Gemini still struggle with.”
Our reading: Classical AI has had only limited success in turning input into checkable logical form and reasoning over it, and systems like Gemini still struggle with it.
No horizon Evidence as of
Limitation: Limited success is a qualitative judgment about general coverage.
Source: No, RAG is probably not going to rescue the current situation
Assessment: UnresolvedSaid
Gary Marcus Observation
“Investors put in over $100 billion into the driverless car industry, and so far have little to show far it.”
Our reading: Investors have put over $100 billion into driverless cars and so far have little to show for it.
No horizon Evidence as of
Limitation: Deployment is not revenue, profit, or investor return.
Source: No, RAG is probably not going to rescue the current situation
Assessment: SupportedSaid
Gary Marcus Observation
“The trouble is that so far RAG has been hit and miss.”
Our reading: So far retrieval-augmented generation has produced inconsistent results.
No horizon Evidence as of
Limitation: Citation correctness is narrower than overall factual accuracy.
Source: No, RAG is probably not going to rescue the current situation
Assessment: UnresolvedSaid
Gary Marcus Forecast
“RAG may reduce hallucinations somewhat, but it certainly won’t come close to eliminating them.”
Our reading: RAG may somewhat reduce hallucinations but will not come close to eliminating them.
No deadline stated Evidence as of
Limitation: The claim supplies no deadline or numerical definition of near-elimination.
Source: No, RAG is probably not going to rescue the current situation
Assessment: UnresolvedSaid
Gary Marcus Observation
“There’s no reason to think that any such thing yet exists, nor that building such a comparison tool is any easier than solving AGI itself.”
Our reading: There is no reason to think that a technology yet exists for comparing an LLM's output with the text of a reliable source and determining with very high accuracy whether they say the same thing, nor that building such a comparison tool is any easier than solving AGI.
No horizon Evidence as of
Limitation: The compound claim combines an empirical existence question with an undefined difficulty comparison.
Source: No, RAG is probably not going to rescue the current situation
Assessment: SupportedSaid
Gary Marcus Observation
“The road from research idea to reliable production remains steep.”
Our reading: Getting RAG from a research idea to reliable production remains difficult.
No horizon Evidence as of
Limitation: The implementation report covers three cases, not a representative industry survey.
Source: No, RAG is probably not going to rescue the current situation
Assessment: SupportedSaid
Gary Marcus Observation
“Reality is that RAG is still in its infancy, a work in progress much as LLMs themselves are, an no more guaranteed to be correct. Like LLMs, RAG can appear spectacular at times, but there is no evidence that it is a cure-all.”
Our reading: RAG is immature, not guaranteed correct, and there is no evidence it is a cure-all for LLM problems.
No horizon Evidence as of
Limitation: The evidence does not give 'infancy' an objective maturity threshold or show that every RAG application was immature.
Source: No, RAG is probably not going to rescue the current situation
Assessment: UnresolvedSaid
Gary Marcus Observation
“Gary Marcus has said from day 1 that LLMs are oversold.”
Our reading: Gary Marcus has said from day 1 that LLMs are oversold.
No horizon Evidence as of
Limitation: The documented criticism concerns GPT-2 and cannot establish his entire earlier record.
Source: Belief in magic may be declining
Assessment: MixedSaid
Gary Marcus Observation
“So let me get this straight? Sora’s can’t reliably handle basic physics, ChatGPT had an unexplained meltdown, Gemini can’t even remember who was on Apollo 11, there are no formal guarantees that any of this will ever work, you yourself admit you don’t know how it works, or how much better it will get next time around, and you want another $7 trillion dollars?”
Our reading: ChatGPT had an unexplained meltdown.
No horizon Evidence as of
Limitation: The incident was initially unexplained. Reading the adjective as referring only to that initial stage would change the assessment.
Source: Belief in magic may be declining
Assessment: UnresolvedSaid
Gary Marcus Observation
“The reality, though is that these systems have never been been stable.”
Our reading: Generative AI systems have never been stable.
No horizon Evidence as of
Limitation: Stable has an ordinary meaning, but its boundary is not specified.
Source: ChatGPT has gone berserk
Assessment: SupportedSaid
Gary Marcus Observation
“Over the last few hours, people reporting having been report a variety of problems with ChatGPT:”
Our reading: Over the last few hours before the post, people reported a variety of problems with ChatGPT.
No horizon Evidence as of
Limitation: The evidence establishes reports, not the prevalence of malfunctions across users.
Source: ChatGPT has gone berserk
Assessment: SupportedSaid
Gary Marcus Observation
“And one wants the system to have a nuanced understanding of what is even been asked. (That’s the clearly lacking now).”
Our reading: Current AI systems clearly lack a nuanced understanding of what is being asked.
No horizon Evidence as of
Limitation: The result depends on reading the claim in its immediate Gemini image-generation context.
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: UnresolvedSaid
Gary Marcus Observation
“Current systems mix historical data, images from old films etc, all together and are way too unsophisticated to determine what would count as representative current sample.”
Our reading: Current AI systems mix historical and current data indiscriminately and are too unsophisticated to determine a representative current sample.
No horizon Evidence as of
Limitation: The demographic experiments predate the target period.
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: UnresolvedSaid
Gary Marcus Observation
“But here’s the thing. The AI we have got now isn’t really up to the job. Getting the delicate balance here right is way outside current AI’s wheelhouse.”
Our reading: Current AI is not capable of getting the balance between cultural sensitivity and historical accuracy right.
No horizon Evidence as of
Limitation: The result depends on retaining the claim's general scope rather than narrowing it to Gemini.
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Getting AI to behave in a culturally-sensitive yet historically accurate way is hard for two reasons, one having to do with data and cultural change, the other with intelligence.”
Our reading: Making AI culturally sensitive yet historically accurate is hard because of skewed data/cultural change and AI's lack of intelligence.
No horizon Evidence as of
Limitation: Google's explanation is a company account rather than an independently reproduced causal investigation.
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: SupportedSaid
Gary Marcus Observation
“It’s a general issue.”
Our reading: Gemini's historically inaccurate diversity in generated images is a general issue, not limited to popes.
No horizon Evidence as of
Limitation: The tests do not measure how frequently the problem occurred.
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: UnresolvedSaid
Gary Marcus Observation
“It should try to clarify if it was unsure of the request. But current AI just ain’t that smart.”
Our reading: Current AI is not smart enough to behave as a sensible system should, for example by telling historical requests from contemporary ones and requests for real photos from requests for fictional characters, and by asking for clarification when unsure of a request.
No horizon Evidence as of
Limitation: Neither item supplies an evaluation matched to the February 2024 deployments and the full set of request distinctions [e6, e7].
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: SupportedSaid
Gary Marcus Observation
“You might have thought that the overactive guardrail problem would have been solved by now, what with billions more invested and ever-bigger models, not to mention the cultural backlash from the right. You’d be wrong.”
Our reading: Despite billions more in investment and bigger models, the overactive guardrail problem in AI chatbots has not been solved as of February 2024.
No horizon Evidence as of
Limitation: These examples establish persistence, not the frequency of excessive refusals across assistants [e1, e2].
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: UnresolvedSaid
Gary Marcus Forecast
“We are about to have a whole generation of children educated by fake videos that are completely plausible to naive audiences yet biologically (or physically or culturally, etc) incorrect.”
Our reading: Soon, a whole generation of children will be educated by AI-generated fake videos that look plausible to naive viewers but are factually incorrect.
No deadline stated Evidence as of
Limitation: The scope words ('whole generation', 'educated by') are hard to measure.
Source: Absolutely amazing ant video augurs intellectual mayhem
Assessment: SupportedSaid
Gary Marcus Forecast
“In short order, with Sora or some opensource counterparts that will emerge, scammers will start to make photorealistic fake videos simply to sell ads.”
Our reading: Open-source counterparts to Sora will emerge.
Settled before its deadline Evidence as of
Limitation: The conclusion depends on interpreting counterparts functionally, without an unstated requirement for Sora-level performance.
Source: Absolutely amazing ant video augurs intellectual mayhem
Assessment: SupportedSaid
Gary Marcus Observation
“… but, um, ants have six legs.”
Our reading: A Sora-generated ant video got the ants' anatomy wrong: it did not show the six legs real ants have.
No horizon Evidence as of
Limitation: The four-leg count comes from a secondary blog post. Nobody in this review watched the video and counted the legs.
Source: Absolutely amazing ant video augurs intellectual mayhem
Assessment: SupportedSaid
Gary Marcus Observation
“In brief, an Air Canada chatbot confabulated (“hallucinated”) an answer – telling a customer that bereavement trips would be refunded.”
Our reading: An Air Canada chatbot hallucinated an answer telling a customer that bereavement trips would be refunded.
No horizon Evidence as of
Limitation: Support depends on reading refunded as including a partial refund and hallucinated as a description of incorrect policy information.
Assessment: UnresolvedSaid
Gary Marcus Observation
“But it’s drawing the wrong generalization.”
Our reading: Sora arguably shows some generalization, but in the monkey-chess case it is drawing the wrong generalization.
No horizon Evidence as of
Limitation: The evidence packet has no independent description or analysis of the original monkey-chess video.
Source: Sora Can’t Handle the Truth
Assessment: SupportedSaid
Gary Marcus Observation
“Gary Marcus is astonished that some people still haven’t recognized that generative AI has struggled mightily when it comes to world models, high level reasoning and factuality, even as evidence for that conclusion continues to pour in, even with data of ever-greater magnitude and sophistication..”
Our reading: Generative AI has struggled mightily with world models, high-level reasoning and factuality, even with ever more data.
No horizon Evidence as of
Limitation: The severity term is qualitative, so the outcome depends partly on the stated reading.
Source: Sora Can’t Handle the Truth
Assessment: UnresolvedSaid
Gary Marcus Observation
“Sora fundamentally revolves around trying to take pixels and reconstruct what pixels might be nearby in space and time. The forest is never perceived apart from the trees, and the chessboard is never perceived apart from its constituent squares.”
Our reading: Sora fundamentally works by reconstructing nearby pixels in space and time rather than perceiving whole objects.
No horizon Evidence as of
Limitation: The cited review appeared after the post and is used only as context about the original model.
Source: Sora Can’t Handle the Truth
Assessment: SupportedSaid
Gary Marcus Conditional forecast
“We will, I am certain, see more systemic glitches as more people have access.”
Our reading: As more people gain access to Sora, more systemic glitches will be found.
Settled before its deadline Evidence as of
Only if: as more people get access
Source: Sora’s Surreal Physics
Assessment: UnresolvedSaid
Gary Marcus Forecast
“More data won’t solve that problem.”
Our reading: More training data will not solve Sora's hallucination-like physics glitches.
No deadline stated Evidence as of
Limitation: The research experiments test models other than OpenAI's Sora and do not establish what all larger datasets could achieve [e1, e2].
Source: Sora’s Surreal Physics
Assessment: UnresolvedSaid
Gary Marcus Forecast
“And importantly, I predict that many will be hard to remedy.”
Our reading: Many of Sora's physics glitches will be hard to fix.
No deadline stated Evidence as of
Limitation: Public development disclosures provide little detail about targeted repair effort.
Source: Sora’s Surreal Physics
Assessment: UnresolvedSaid
Gary Marcus Observation
“If the system really is being “programmed in English” it is not always doing what is being asked to do:”
Our reading: If ChatGPT/GPT-4 really is being "programmed in English" through its allegedly leaked system prompt, it is not always doing what it is asked to do.
No horizon Evidence as of
Limitation: An absent authentication record does not show that the alleged prompt was fabricated.
Source: The hottest new programming language is English! Or maybe not.
Assessment: SupportedSaid
Gary Marcus Observation
“But the truth is it doesn’t work that well yet.”
Our reading: Programming in English with LLMs does not yet work very well.
No horizon Evidence as of
Limitation: The competitive-programming evaluation does not establish a representative success rate for everyday programming or direct control through English prompts [e1].
Source: The hottest new programming language is English! Or maybe not.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Much of what Reid Southen, Katie Conrad, and I recently described is in flagrant violation of those directives.”
Our reading: Much of what Marcus, Reid Southen and Katie Conrad recently described is in flagrant violation of the directives in ChatGPT's allegedly leaked system prompt.
No horizon Evidence as of
Limitation: This assessment concerns compliance with instructions, not whether an image legally infringes copyright.
Source: The hottest new programming language is English! Or maybe not.
Assessment: UnresolvedSaid
Gary Marcus Observation
“One winner out of three to be exact.”
Our reading: After several retries, Gemini produced only one correct horse-riding-astronaut image out of three.
No horizon Evidence as of
Limitation: This concerns one historical response, not Gemini's general success rate.
Source: Horse rides Astronaut, redux
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“In other words, the problem was with its language understanding, rather than with illustration per se. The model was perfectly capable of drawing a horse riding an astronaut, it just didn’t know that the phrase horse rides astronaut was a request for a horse riding an astronaut.”
Our reading: DALL-E's failure on 'horse rides astronaut' was due to deficient language understanding, not illustration ability.
No horizon Evidence as of
Limitation: The original images and complete sampling records were not available in the retrieved text.
Source: Horse rides Astronaut, redux
Assessment: UnresolvedSaid
Gary Marcus Observation
“I think we can safely reach the same conclusion as before. Gemini can draw the desired image. But it’s still like pulling teeth to get there.”
Our reading: Gemini can draw a horse riding an astronaut, but getting it to do so is very difficult.
No horizon Evidence as of
Limitation: A verified difficult session would support the local observation more strongly than a general claim about typical performance.
Source: Horse rides Astronaut, redux
Assessment: ContradictedSaid
Gary Marcus Observation
“In some ways we have made progress, but in the ones I keep harping on—factuality and compositionality—we have not.”
Our reading: Generative AI has made no progress on factuality and compositionality since 2022, despite progress in other respects.
No horizon Evidence as of
Limitation: The verdict depends on reading progress as measurable improvement rather than complete resolution.
Source: Horse rides Astronaut, redux
Assessment: UnresolvedSaid
Gary Marcus Observation
“In our own brief experiments with GPT-3 (OpenAI has refused us proper scientific access for over a year) we found cases like the following, which reflects a complete failure to understand human biology.”
Our reading: In Marcus and Davis's experiments, GPT-3 produced outputs showing a complete failure to understand human biology.
No horizon Evidence as of
Limitation: The local-error reading is narrower than a claim that GPT-3 understood nothing about biology.
Source: Statistics versus Understanding: The Essence of What Ails Generative AI
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“That’s why statistically improbable requests like horses riding astronauts have always been challenging for generative image systems.”
Our reading: Because they rely on statistics instead of understanding, generative image systems have always struggled with statistically improbable requests such as horses riding astronauts.
No horizon Evidence as of
Limitation: The most directly relevant study concerns one model.
Source: Statistics versus Understanding: The Essence of What Ails Generative AI
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Once again statistical frequency (most guitarists play righthanded) won out over linguistic understanding, to the consternation of Hendrix fans everywhere.”
Our reading: In Wyatt Walls's left-handed guitarist example, statistical frequency (most guitarists play right-handed) once again won out over linguistic understanding.
No horizon Evidence as of
Limitation: Image-based tweets could not be inspected.
Source: Statistics versus Understanding: The Essence of What Ails Generative AI
Assessment: SupportedSaid
Gary Marcus Observation
“And, let’s be honest folks, the “alignment problem” isn’t remotely solved.”
Our reading: The AI alignment problem is not remotely solved.
No horizon Evidence as of
Limitation: The decisive study was a preprint using deliberately constructed backdoors.
Source: Seven reasons why the world should say No to Sam Altman
Assessment: UnresolvedSaid
Gary Marcus Observation
“Negative externalities, essentially none of which OpenAI is absorbing, like the cost of dis- and misinformation, cybercrimes, and the latest, fake books, a scam that has recently hurt at least three authors I know over the last two months.”
Our reading: OpenAI absorbs essentially none of the negative externalities of its products, such as misinformation, cybercrime, and fake books.
No horizon Evidence as of
Limitation: Private payments and remediation costs may not be publicly disclosed.
Source: Seven reasons why the world should say No to Sam Altman
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“They are already a target for attack, and as long as they are successful they will continue to be; if they have direct access to user devices, the risks of them getting hacked will increase, and the spoils of war for the attackers will increase, since it enables essentially infinite mischief (e.g., draining bank balances, stealing credit card numbers, placing fraudulent transactions, extortion based on private photos, holding devices for ransom, and so on).”
Our reading: If OpenAI gains direct access to user devices, the risk of it being hacked and the rewards to attackers will increase.
No deadline stated Evidence as of
Only if: if OpenAI has direct access to user devices
Source: What Could Possibly Go Wrong with Sam Altman’s New Ambitions?
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Chatbots are struggling to make the money people envision, so he is doubling down to (a) agents, and (b) a multitrillion-dollar chips-and-energy infrastructure play.”
Our reading: Chatbots are struggling to make the money people envision, so Altman is doubling down on agents and on a multitrillion-dollar chips-and-energy infrastructure play.
No horizon Evidence as of
Limitation: The financial measure and expected result are unspecified.
Source: What Could Possibly Go Wrong with Sam Altman’s New Ambitions?
Assessment: SupportedSaid
Gary Marcus Causal claim
“In a system that can write emails, make appointments, sign checks, etc, the consequences of unreliability, discomprehension, and hallucination all escalate.”
Our reading: When LLMs can take actions like writing emails or signing checks, the consequences of their unreliability and hallucinations escalate.
No horizon Evidence as of
Limitation: The evidence supports increased potential consequences, not inevitable harm from every action.
Source: What Could Possibly Go Wrong with Sam Altman’s New Ambitions?
Assessment: SupportedSaid
Gary Marcus Forecast
“This list is probably not exhaustive and will continue to grow.”
Our reading: The list of OpenAI/LLM security issues will continue to grow.
Settled before its deadline Evidence as of
Limitation: This establishes additional vulnerability disclosure, not a rate of future incidents.
Source: What Could Possibly Go Wrong with Sam Altman’s New Ambitions?
Assessment: SupportedSaid
Gary Marcus Observation
“More evidence against his claims just keeps piling up.”
Our reading: Evidence against Hinton's claims that GenAI understands and doesn't store things keeps accumulating.
No horizon Evidence as of
Limitation: The result depends on reading the statement as an evidentiary observation rather than a conclusive theory of understanding.
Source: Further Trouble in Hinton City
Assessment: UnresolvedSaid
Gary Marcus Observation
“GenAI systems do effectively store things. (In this case, canonical video game controller images.)”
Our reading: GenAI systems effectively store things such as canonical video game controller images.
No horizon Evidence as of
Limitation: If the controller parenthetical is treated only as an illustration, the general storage proposition is supported.
Source: Further Trouble in Hinton City
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Noncanonical situations continue to remain difficult for generative AI systems, precisely because frequent statistics are a mediocre substitute for deep understanding.”
Our reading: Generative AI systems still struggle with noncanonical situations because frequency statistics are a poor substitute for deep understanding.
No horizon Evidence as of
Limitation: Both studies predate the target period and cover narrower image-generation settings.
Source: Further Trouble in Hinton City
Assessment: UnresolvedSaid
Gary Marcus Observation
“They (often) don’t come up with a novel independent design that would meet the requirements.”
Our reading: GenAI systems often don't come up with a novel independent design that would meet the requirements.
No horizon Evidence as of
Limitation: The packet lacks representative evaluations jointly measuring novelty and functional compliance.
Source: Further Trouble in Hinton City
Assessment: SupportedSaid
Gary Marcus Observation
“Reliability remain a problem.”
Our reading: Reliability remains a problem for large language models.
No horizon Evidence as of
Limitation: Benchmark performance varies by task and scoring method.
Assessment: SupportedSaid
Gary Marcus Observation
“Scale isn’t everything, and hasn’t and won’t be enough to solve any of the problems I just listed above.”
Our reading: Scaling has not been enough to solve the problems of understanding, compositionality, negation, and reliability in LLMs.
No horizon Evidence as of
Limitation: The result depends on treating understanding as observable interpretation and generalization.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Scale isn’t everything, and hasn’t and won’t be enough to solve any of the problems I just listed above.”
Our reading: Scaling will not be enough to solve the problems of understanding, compositionality, negation, and reliability in LLMs.
No deadline stated Evidence as of
Limitation: The boundary between scaling and a new method is unclear.
Assessment: SupportedSaid
Gary Marcus Observation
“Compositionality remains a problem. So does negation.”
Our reading: Compositionality and negation remain problems for large language models.
No horizon Evidence as of
Limitation: Results depend on task, model version and prompt.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“But anyone who fears that GPT-5 is going to convey an immense advantage in military strategy has got to be kidding.”
Our reading: GPT-5 will not convey an immense advantage in military strategy.
No deadline stated Evidence as of
Limitation: Public evidence may omit classified deployments.
Assessment: MixedSaid
Gary Marcus Observation
“Special-purpose AI has been great for static, easily-simulated games with rules that have been stable for centuries, like Chess, Go, and even Diplomacy, but never been great at dynamic environments in which the environment itself is unpredictable.”
Our reading: Special-purpose AI has been great for static, easily simulated games with rules that have been stable for centuries, such as chess, Go and even Diplomacy, but has never been great at dynamic environments in which the environment itself is unpredictable.
No horizon Evidence as of
Limitation: The distinction between environmental uncertainty and unrestricted task change affects the result.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Even with a lot of data, GPT-4 is playing chess at around the level where (top) chess programs were in 1976, not even close to current state of the art.”
Our reading: GPT-4 plays chess at about the level of top chess programs in 1976, far from the current state of the art.
No horizon Evidence as of
Limitation: The 1976 tournament figure comes from a secondary account [e2].
Assessment: SupportedSaid
Gary Marcus Observation
“It’s notorious for making illegal moves in chess.”
Our reading: ChatGPT is notorious for making illegal moves in chess.
No horizon Evidence as of
Limitation: Notoriety is qualitative.
Assessment: MixedSaid
Gary Marcus Observation
“The New York Times versus OpenAI lawsuit made that abundantly clear, with a hundred examples like this one, in which ChatGPT reconstructed entire stretches of text (red) from brief prompts2 (black text in top line):”
Our reading: The NYT v. OpenAI lawsuit showed about a hundred examples of ChatGPT reconstructing entire stretches of text from brief prompts.
No horizon Evidence as of
Limitation: Both filings are party submissions, not judicial findings about experimental validity.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
““Given vast datasets, LLMs approximate well, but their understanding is at best superficial. That’s *why* they are unreliable, and unstable, hallucinate, are constitutionally unable to fact check, etc.””
Our reading: Given vast datasets, LLMs approximate well, but their understanding is at best superficial, and that is why they are unreliable, unstable, hallucinate, are constitutionally unable to fact-check, and so on.
No horizon Evidence as of
Limitation: Partial error detection is not reliable verification of arbitrary claims.
Assessment: SupportedSaid
Gary Marcus Observation
“But the variety of the examples along with the technical literature on memorization and data leakage in large language models make clear that this kind of reconstruction certainly can happen with some regularity.”
Our reading: Reconstruction of training text by large language models can happen with some regularity.
No horizon Evidence as of
Limitation: Marcus supplies no numerical meaning for 'some regularity'.
Assessment: SupportedSaid
Gary Marcus Observation
“LLMs don’t store things in simple databases, but when presented with enough data, their regeneration of distributed bits has demonstrably duplicated lengthy training materials multiple times, a phenomenon that is called memorization in the field.”
Our reading: Given enough data, LLMs have demonstrably duplicated lengthy training materials multiple times (memorization).
No horizon Evidence as of
Limitation: Extraction methods can involve deliberate prompting and specialized decoding.
Assessment: UnresolvedSaid
Gary Marcus Observation
“OpenAI will no doubt by now have filtered this specific example so that it can no longer be replicated, now that it has appeared in a lawsuit.”
Our reading: OpenAI has no doubt by now filtered this specific NYT-lawsuit example so that it can no longer be replicated, now that it has appeared in a lawsuit.
No horizon Evidence as of
Limitation: The relevant generation setup is unspecified.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Because of the immense amounts of data they are now trained on, and because of the immense number of parameters on which they are trained, that reconstruction can in fact sometimes come very close to memorization.”
Our reading: Because of their huge training data and parameter counts, LLM reconstruction of inputs sometimes comes very close to memorization.
No horizon Evidence as of
Limitation: This outcome concerns the full causal sentence, not whether LLMs memorize.
Assessment: UnresolvedSaid
Gary Marcus Observation
“LLMs do (sometimes) regurgitate large chunks of effectively stored text, and they don’t understand anything at a deep level.”
Our reading: LLMs sometimes regurgitate large chunks of effectively stored text and do not understand anything at a deep level.
No horizon Evidence as of
Limitation: Closed-model training membership is inferred from matching pre-existing material rather than inspection of the complete private training corpus.
Assessment: SupportedSaid
Gary Marcus Observation
“Neural networks do effectively memorize some of what they are trained on, and indeed there is a whole subfield now about the topic.”
Our reading: Neural networks effectively memorize some of what they are trained on, and there is now a whole subfield devoted to this topic.
No horizon Evidence as of
Limitation: For closed models, the researchers used an auxiliary corpus because the complete training data were unavailable.
Assessment: SupportedSaid
Gary Marcus Observation
“An entire cast of deepfaked people include a deep faked CFO scammed a company out $25 million.”
Our reading: Deepfaked participants, including a deepfaked CFO, on a video call scammed a company out of $25 million.
No horizon Evidence as of
Limitation: Support depends on reading $25 million as approximate, not exact.
Source: Deepfaked Shit is Getting Real
Assessment: SupportedSaid
Gary Marcus Observation
“Even scarier is that the employee who was tricked was suspicious, not naive, and still fell for it:”
Our reading: The deepfake scam worked even on an employee who was suspicious rather than naive.
No horizon Evidence as of
Limitation: The account of suspicion comes through police reporting, rather than a direct employee interview.
Source: Deepfaked Shit is Getting Real
Assessment: SupportedSaid
Gary Marcus Observation
“I am here to tell you that OpenAI’s research, analyzed carefully, shows no such thing, and that there is plenty of reason to be concerned.”
Our reading: OpenAI's bioweapons research, analyzed carefully, does not show that GPT-4 poses at most a slight risk of helping create biological threats, and there is plenty of reason to be concerned.
No horizon Evidence as of
Limitation: OpenAI itself acknowledged uncertainty; Marcus's criticism should not imply that the study concealed all these limitations.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: SupportedSaid
Gary Marcus Observation
“For one thing, in the expert population, the effect is in the predicted direction for all five tasks (ideation, acquisition, magnification, formulation, and release).”
Our reading: In OpenAI's threshold analysis, among experts the effect was in the predicted direction for all five tasks (ideation, acquisition, magnification, formulation and release).
No horizon Evidence as of
Limitation: These observed threshold proportions alone do not establish statistical significance or real-world biological threat capability.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: SupportedSaid
Gary Marcus Observation
“In fact, if you simply collapsed all the measurements of accuracy, and did the single most obvious test here, a simple t-test, the results would (as Footnote C implies) be significant.”
Our reading: If all the accuracy measurements in OpenAI's study were collapsed and tested with a simple t-test, the results would be statistically significant, as Footnote C implies.
No horizon Evidence as of
Limitation: The result does not establish that this analysis was uniquely appropriate or that multiple-comparison adjustment was unnecessary.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons