Supported claims, page 4 of 6
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Supported claims (531)
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Page 4 of 6: claims 301 to 400 of 531, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Gary Marcus Observation
“For one thing, the bad press about Sam Altman and OpenAI, who once seemingly could do no wrong, just keeps coming.”
Our reading: Negative press coverage of Sam Altman and OpenAI, once seen as able to do no wrong, keeps coming.
No horizon Evidence as of
Limitation: This is a finding about the existence of repeated coverage, not its prevalence or rate of increase.
Source: AI as Reality Television
Assessment: SupportedSaid
Gary Marcus Observation
“For all the daily claims of “exponential progress”, and for all the tens of billions that have been invested since, reliability is still a dream, and of course toxicity and hallucinations are still with us.”
Our reading: Despite tens of billions of dollars invested since 2022, deep learning systems are still unreliable and still produce toxicity and hallucinations.
No horizon Evidence as of
Limitation: The reliability phrase is rhetorical and supplies no threshold.
Assessment: SupportedSaid
Gary Marcus Observation
“Many people are reaching essential the same perceptions.”
Our reading: Many people are reaching essentially the same perceptions of Sam Altman.
No horizon Evidence as of
Limitation: The undefined word 'many' makes the result depend on a qualitative reading.
Source: Sam Altman’s fall from grace
Assessment: SupportedSaid
Gary Marcus Observation
“In November, the narrative was about how everybody loved Sam. Things have changed. A lot.”
Our reading: The narrative about Sam Altman, which in November was that everybody loved him, has changed a lot.
No horizon Evidence as of
Limitation: The result depends on reading 'narrative' as prominent coverage and commentary, rather than measured population sentiment.
Source: Sam Altman’s fall from grace
Assessment: SupportedSaid
Gary Marcus Causal claim
“The very process that LLMs use to generalize also creates hallucinations.”
Our reading: The same process LLMs use to generalize is what creates their hallucinations.
No horizon Evidence as of
Limitation: The theory covers rare, arbitrary facts most clearly. Other kinds of hallucination may have other causes.
Assessment: SupportedSaid
Gary Marcus Forecast
“LLMs are well on their way to becoming a commodity; price wars will be intense.”
Our reading: LLMs are becoming a commodity and price wars among providers will be intense.
Settled before its deadline Evidence as of
Limitation: Commoditization is assessed as a direction of travel, consistent with well on their way.
Source: Groupthink versus Tulips. What will historians think of 2024?
Assessment: SupportedSaid
Gary Marcus Observation
“Adding visual data to LLMs hasn’t solved hallucinations.”
Our reading: Adding visual data to LLMs has not solved hallucinations.
No horizon Evidence as of
Limitation: 'Solved' is not defined. The claim is read as hallucinations remaining a material problem, not as adding vision bringing no improvement at all. The evidence packet notes that the GPT-4V system card reports fewer hallucinations during a beta, but that passage is not in the cited excerpt.
Source: TED, AI, and the test of time
Assessment: SupportedSaid
Gary Marcus Observation
“What it actually shows is some improvements, mostly modest, over the last few months for a bunch of different measures.”
Our reading: OpenAI's GPT-4 Turbo graph shows only mostly modest improvements over recent months.
No horizon Evidence as of
Limitation: Modest is a qualitative judgment, and relative changes can look larger.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Gary Marcus Observation
“I think it’s fair to say that they have not. GPT-Turbo released this week still produces its share of groaners, like this interchange Phil Libin just sent:”
Our reading: Hallucinations and stupid errors in LLMs have not been solved; GPT-4 Turbo still makes them.
No horizon Evidence as of
Limitation: The coding test measures task completion, not hallucination frequency.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Gary Marcus Observation
“Already in recent weeks, Inflection AI largely closed shop, Stability AI is struggling, and LLM-based autonomous vehicle company called Ghost closed shop, and a software engineer on YouTube raised quite serious questions about the wildly-hyped AI coding system Devin.”
Our reading: In recent weeks Inflection AI largely closed, Stability AI is struggling, Ghost closed, and serious questions were raised about Devin.
No horizon Evidence as of
Limitation: The Inflection finding depends on the meaning of 'largely'; the company continued operating.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Gary Marcus Observation
“Huge jump from GPT-2 to GPT-3. Huge jump from GPT-3 to GPT-4 … and not so much for GPT-4 (13 months ago) to GPT-4 Turbo (just released). It’s hard not to see this plot as tentative evidence for the hypothesis of diminishing returns.”
Our reading: MMLU gains from GPT-4 to GPT-4 Turbo are small compared with earlier generational jumps, tentatively evidencing diminishing returns.
No horizon Evidence as of
Limitation: The records do not compare all four models under one protocol: GPT-2 was fine-tuned, GPT-3 used few-shot prompting, GPT-4 used five-shot prompting, and Turbo used zero-shot chain-of-thought prompting [e1, e2, e3].
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: SupportedSaid
Gary Marcus Observation
“Tesla FSD is getting better. But is getting enough better?”
Our reading: Tesla FSD is getting better, and he raises the question of whether it is getting enough better.
No horizon Evidence as of
Limitation: The tracker figures in e1 come from crowdsourced, self-reported data. e1 notes that data for later v12.3.x releases still needed weeks to be entered.
Source: 24 hours in AI news
Assessment: SupportedSaid
Gary Marcus Observation
“current models aren’t literally look up tables; they can generalize to some degree, but not enough.”
Our reading: Current AI models can generalize to some degree but not enough.
No horizon Evidence as of
Limitation: The outcome depends on interpreting 'not enough' through the surrounding discussion of reliability.
Assessment: SupportedSaid
Gary Marcus Observation
“Neural networks (at least in the configurations that have been dominant over the last three decades) have trouble generalizing beyond the multidimensional space that surrounds their training examples.”
Our reading: Neural networks in the configurations dominant for the last three decades have trouble generalizing beyond the space surrounding their training examples.
No horizon Evidence as of
Limitation: The outcome depends on reading the spatial language as unfamiliar conditions rather than a precise geometric boundary.
Assessment: SupportedSaid
Gary Marcus Observation
“The good news is that word is finally getting out more broadly.”
Our reading: Awareness of the danger generative AI poses to the information ecosystem is finally spreading more widely.
No horizon Evidence as of
Limitation: The trend ('more broadly') is inferred from high levels, not from a measured before-and-after comparison.
Assessment: SupportedSaid
Gary Marcus Observation
“Meanwhile X itself is promoting nonsense.”
Our reading: X itself is promoting nonsense.
No horizon Evidence as of
Limitation: The assessment relies on direct news reporting with screenshots and an independent fact-check; no separately retrievable original X display was obtained.
Assessment: SupportedSaid
Gary Marcus Causal claim
“Ironic that his own chatbot, Grok, is making the problem worse.”
Our reading: Musk's own chatbot, Grok, is making the information-pollution problem worse.
No horizon Evidence as of
Limitation: The false Iran story began with user posts; Grok's added effect on spread is not separately measured.
Assessment: SupportedSaid
Gary Marcus Observation
“The bad news, which outweighs the good, is that the problem is quickly getting worse.”
Our reading: Generative-AI pollution of the information ecosystem is getting worse quickly as of April 2024.
No horizon Evidence as of
Limitation: Tracker counts come from one commercial source and may partly reflect more searching rather than only more sites.
Assessment: SupportedSaid
Gary Marcus Observation
“The trouble is that the bigger the model, the higher the internal costs. And current-sized models, already expensive, aren’t reliable enough for many jobs.”
Our reading: Bigger models have higher costs, and current-sized models are already expensive and not reliable enough for many jobs.
No horizon Evidence as of
Limitation: The result depends on comparing otherwise similar implementations and interpreting reliability as suitability for unverified use.
Assessment: SupportedSaid
Gary Marcus Observation
“As noted here before, one study indicates that code quality is going down.”
Our reading: One study indicates code quality is going down amid AI-assisted coding.
No horizon Evidence as of
Limitation: The study uses indirect quality indicators.
Source: The race between positive and negative applications of Generative AI is on – and not looking pretty
Assessment: SupportedSaid
Gary Marcus Observation
“On the other hand, the use cases on the negative side of the ledger are proliferating fast, literally faster than I can report them here, from fake books (undermining authors) to rings of fake websites designed to sell ads”
Our reading: Harmful uses of generative AI (fake books, ad-farm websites, election misinformation, polluted science) are proliferating rapidly.
No horizon Evidence as of
Limitation: Support depends on reading proliferation as broad expansion rather than proven rapid growth in every category.
Source: The race between positive and negative applications of Generative AI is on – and not looking pretty
Assessment: SupportedSaid
Gary Marcus Observation
“Some core problems have lingered, particularly hallucinations, instability and occasional nonsensical errors.”
Our reading: Hallucinations, instability and occasional nonsensical errors remain unsolved in GPT-4 class LLMs.
No horizon Evidence as of
Limitation: The research tests selected tasks rather than representative everyday usage.
Source: Happy 1st Birthday, GPT-4
Assessment: SupportedSaid
Gary Marcus Forecast
“We can expect price wars.”
Our reading: There will be price wars among LLM providers.
Settled before its deadline Evidence as of
Limitation: The decisive episode occurred in China; the claim supplied no geographic restriction.
Source: Happy 1st Birthday, GPT-4
Assessment: SupportedSaid
Gary Marcus Observation
“Translation: But (revealed a few weeks later) behind the scenes we will try hard to water down your regulations.”
Our reading: While publicly asking the Senate to regulate it, OpenAI worked behind the scenes to water down regulations.
No horizon Evidence as of
Limitation: The documented request concerned the EU AI Act; it does not establish efforts to weaken a U.S. Senate proposal [e2].
Source: OpenAI’s Lies and Half-Truths
Assessment: SupportedSaid
Gary Marcus Observation
“GenAI is messing with science.”
Our reading: Generative AI is currently damaging science.
No horizon Evidence as of
Limitation: The conclusion concerns an identifiable publication failure, not the frequency or aggregate consequences of such failures.
Source: The Year of the Big-Balled Rat
Assessment: SupportedSaid
Gary Marcus Observation
“And the apparent copyright infringements that Reid Southen and I discussed in early January continue apace two months later, as I found in a quick experiment this morning:”
Our reading: The apparent copyright infringements that Marcus and Reid Southen discussed in early January 2024 were continuing apace two months later, based on a quick experiment Marcus ran on the morning of the post.
No horizon Evidence as of
Limitation: The result depends on reading 'apace' as continued ready production rather than a quantitative rate claim.
Source: OpenAI’s “Own Goal”
Assessment: SupportedSaid
Gary Marcus Observation
“Almost nothing is known about the precise internal mechanisms of GPT-4 or what it has trained on.”
Our reading: Almost nothing is known about the precise internal mechanisms of GPT-4 or what it has been trained on.
No horizon Evidence as of
Limitation: Official nondisclosure does not prove that outsiders knew nothing.
Source: OpenAI’s “Own Goal”
Assessment: SupportedSaid
Gary Marcus Observation
“They knew they were misleading people and failed to take an opportunity to correct.”
Our reading: OpenAI knew it was misleading people into believing it meant open-sourced AI and failed to take an opportunity, at a May 2016 meeting, to correct that belief.
No horizon Evidence as of
Limitation: The supplied evidence consists of one reported account; no independent recording, transcript, or meeting minutes are supplied.
Source: OpenAI’s “Own Goal”
Assessment: SupportedSaid
Gary Marcus Observation
“On overt measures, the systems were fine. On covert measures, they were a disaster:”
Our reading: LLMs tested looked fine on overt racism measures but did very badly on covert racism measures when prompted with African American English.
No horizon Evidence as of
Limitation: The abstract does not establish that every tested model was free of overt bias.
Source: Covert racism in LLMs
Assessment: SupportedSaid
Gary Marcus Observation
“Dropped is any mention of (a) the nonprofit (even though legally the company remains a nonprofit), (b) open source, once central both to their mission and name, and (c) the original phrasing about being "unconstrained by a need to generate financial return."”
Our reading: OpenAI's current homepage mission statement no longer mentions the nonprofit, open source, or being unconstrained by a need to generate financial return.
No horizon Evidence as of
Limitation: The assessment covers text preserved and extracted from the archive, not unextracted video or image content.
Source: The Irony and the Agony of Elon Musk's Lawsuit against OpenAI
Assessment: SupportedSaid
Gary Marcus Observation
“That’s exactly what classical AI has been trying to do for decades, with only limited success. And what systems like Gemini still struggle with.”
Our reading: Classical AI has had only limited success in turning input into checkable logical form and reasoning over it, and systems like Gemini still struggle with it.
No horizon Evidence as of
Limitation: Limited success is a qualitative judgment about general coverage.
Source: No, RAG is probably not going to rescue the current situation
Assessment: SupportedSaid
Gary Marcus Observation
“The trouble is that so far RAG has been hit and miss.”
Our reading: So far retrieval-augmented generation has produced inconsistent results.
No horizon Evidence as of
Limitation: Citation correctness is narrower than overall factual accuracy.
Source: No, RAG is probably not going to rescue the current situation
Assessment: SupportedSaid
Gary Marcus Observation
“The road from research idea to reliable production remains steep.”
Our reading: Getting RAG from a research idea to reliable production remains difficult.
No horizon Evidence as of
Limitation: The implementation report covers three cases, not a representative industry survey.
Source: No, RAG is probably not going to rescue the current situation
Assessment: SupportedSaid
Gary Marcus Observation
“Reality is that RAG is still in its infancy, a work in progress much as LLMs themselves are, an no more guaranteed to be correct. Like LLMs, RAG can appear spectacular at times, but there is no evidence that it is a cure-all.”
Our reading: RAG is immature, not guaranteed correct, and there is no evidence it is a cure-all for LLM problems.
No horizon Evidence as of
Limitation: The evidence does not give 'infancy' an objective maturity threshold or show that every RAG application was immature.
Source: No, RAG is probably not going to rescue the current situation
Assessment: SupportedSaid
Gary Marcus Observation
“Over the last few hours, people reporting having been report a variety of problems with ChatGPT:”
Our reading: Over the last few hours before the post, people reported a variety of problems with ChatGPT.
No horizon Evidence as of
Limitation: The evidence establishes reports, not the prevalence of malfunctions across users.
Source: ChatGPT has gone berserk
Assessment: SupportedSaid
Gary Marcus Observation
“And one wants the system to have a nuanced understanding of what is even been asked. (That’s the clearly lacking now).”
Our reading: Current AI systems clearly lack a nuanced understanding of what is being asked.
No horizon Evidence as of
Limitation: The result depends on reading the claim in its immediate Gemini image-generation context.
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: SupportedSaid
Gary Marcus Observation
“It’s a general issue.”
Our reading: Gemini's historically inaccurate diversity in generated images is a general issue, not limited to popes.
No horizon Evidence as of
Limitation: The tests do not measure how frequently the problem occurred.
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: SupportedSaid
Gary Marcus Observation
“You might have thought that the overactive guardrail problem would have been solved by now, what with billions more invested and ever-bigger models, not to mention the cultural backlash from the right. You’d be wrong.”
Our reading: Despite billions more in investment and bigger models, the overactive guardrail problem in AI chatbots has not been solved as of February 2024.
No horizon Evidence as of
Limitation: These examples establish persistence, not the frequency of excessive refusals across assistants [e1, e2].
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: SupportedSaid
Gary Marcus Forecast
“In short order, with Sora or some opensource counterparts that will emerge, scammers will start to make photorealistic fake videos simply to sell ads.”
Our reading: Open-source counterparts to Sora will emerge.
Settled before its deadline Evidence as of
Limitation: The conclusion depends on interpreting counterparts functionally, without an unstated requirement for Sora-level performance.
Source: Absolutely amazing ant video augurs intellectual mayhem
Assessment: SupportedSaid
Gary Marcus Observation
“… but, um, ants have six legs.”
Our reading: A Sora-generated ant video got the ants' anatomy wrong: it did not show the six legs real ants have.
No horizon Evidence as of
Limitation: The four-leg count comes from a secondary blog post. Nobody in this review watched the video and counted the legs.
Source: Absolutely amazing ant video augurs intellectual mayhem
Assessment: SupportedSaid
Gary Marcus Observation
“In brief, an Air Canada chatbot confabulated (“hallucinated”) an answer – telling a customer that bereavement trips would be refunded.”
Our reading: An Air Canada chatbot hallucinated an answer telling a customer that bereavement trips would be refunded.
No horizon Evidence as of
Limitation: Support depends on reading refunded as including a partial refund and hallucinated as a description of incorrect policy information.
Assessment: SupportedSaid
Gary Marcus Observation
“Gary Marcus is astonished that some people still haven’t recognized that generative AI has struggled mightily when it comes to world models, high level reasoning and factuality, even as evidence for that conclusion continues to pour in, even with data of ever-greater magnitude and sophistication..”
Our reading: Generative AI has struggled mightily with world models, high-level reasoning and factuality, even with ever more data.
No horizon Evidence as of
Limitation: The severity term is qualitative, so the outcome depends partly on the stated reading.
Source: Sora Can’t Handle the Truth
Assessment: SupportedSaid
Gary Marcus Conditional forecast
“We will, I am certain, see more systemic glitches as more people have access.”
Our reading: As more people gain access to Sora, more systemic glitches will be found.
Settled before its deadline Evidence as of
Only if: as more people get access
Source: Sora’s Surreal Physics
Assessment: SupportedSaid
Gary Marcus Observation
“But the truth is it doesn’t work that well yet.”
Our reading: Programming in English with LLMs does not yet work very well.
No horizon Evidence as of
Limitation: The competitive-programming evaluation does not establish a representative success rate for everyday programming or direct control through English prompts [e1].
Source: The hottest new programming language is English! Or maybe not.
Assessment: SupportedSaid
Gary Marcus Observation
“And, let’s be honest folks, the “alignment problem” isn’t remotely solved.”
Our reading: The AI alignment problem is not remotely solved.
No horizon Evidence as of
Limitation: The decisive study was a preprint using deliberately constructed backdoors.
Source: Seven reasons why the world should say No to Sam Altman
Assessment: SupportedSaid
Gary Marcus Causal claim
“In a system that can write emails, make appointments, sign checks, etc, the consequences of unreliability, discomprehension, and hallucination all escalate.”
Our reading: When LLMs can take actions like writing emails or signing checks, the consequences of their unreliability and hallucinations escalate.
No horizon Evidence as of
Limitation: The evidence supports increased potential consequences, not inevitable harm from every action.
Source: What Could Possibly Go Wrong with Sam Altman’s New Ambitions?
Assessment: SupportedSaid
Gary Marcus Forecast
“This list is probably not exhaustive and will continue to grow.”
Our reading: The list of OpenAI/LLM security issues will continue to grow.
Settled before its deadline Evidence as of
Limitation: This establishes additional vulnerability disclosure, not a rate of future incidents.
Source: What Could Possibly Go Wrong with Sam Altman’s New Ambitions?
Assessment: SupportedSaid
Gary Marcus Observation
“More evidence against his claims just keeps piling up.”
Our reading: Evidence against Hinton's claims that GenAI understands and doesn't store things keeps accumulating.
No horizon Evidence as of
Limitation: The result depends on reading the statement as an evidentiary observation rather than a conclusive theory of understanding.
Source: Further Trouble in Hinton City
Assessment: SupportedSaid
Gary Marcus Observation
“Reliability remain a problem.”
Our reading: Reliability remains a problem for large language models.
No horizon Evidence as of
Limitation: Benchmark performance varies by task and scoring method.
Assessment: SupportedSaid
Gary Marcus Observation
“Scale isn’t everything, and hasn’t and won’t be enough to solve any of the problems I just listed above.”
Our reading: Scaling has not been enough to solve the problems of understanding, compositionality, negation, and reliability in LLMs.
No horizon Evidence as of
Limitation: The result depends on treating understanding as observable interpretation and generalization.
Assessment: SupportedSaid
Gary Marcus Observation
“Compositionality remains a problem. So does negation.”
Our reading: Compositionality and negation remain problems for large language models.
No horizon Evidence as of
Limitation: Results depend on task, model version and prompt.
Assessment: SupportedSaid
Gary Marcus Observation
“It’s notorious for making illegal moves in chess.”
Our reading: ChatGPT is notorious for making illegal moves in chess.
No horizon Evidence as of
Limitation: Notoriety is qualitative.
Assessment: SupportedSaid
Gary Marcus Observation
“But the variety of the examples along with the technical literature on memorization and data leakage in large language models make clear that this kind of reconstruction certainly can happen with some regularity.”
Our reading: Reconstruction of training text by large language models can happen with some regularity.
No horizon Evidence as of
Limitation: Marcus supplies no numerical meaning for 'some regularity'.
Assessment: SupportedSaid
Gary Marcus Observation
“LLMs don’t store things in simple databases, but when presented with enough data, their regeneration of distributed bits has demonstrably duplicated lengthy training materials multiple times, a phenomenon that is called memorization in the field.”
Our reading: Given enough data, LLMs have demonstrably duplicated lengthy training materials multiple times (memorization).
No horizon Evidence as of
Limitation: Extraction methods can involve deliberate prompting and specialized decoding.
Assessment: SupportedSaid
Gary Marcus Observation
“Neural networks do effectively memorize some of what they are trained on, and indeed there is a whole subfield now about the topic.”
Our reading: Neural networks effectively memorize some of what they are trained on, and there is now a whole subfield devoted to this topic.
No horizon Evidence as of
Limitation: For closed models, the researchers used an auxiliary corpus because the complete training data were unavailable.
Assessment: SupportedSaid
Gary Marcus Observation
“An entire cast of deepfaked people include a deep faked CFO scammed a company out $25 million.”
Our reading: Deepfaked participants, including a deepfaked CFO, on a video call scammed a company out of $25 million.
No horizon Evidence as of
Limitation: Support depends on reading $25 million as approximate, not exact.
Source: Deepfaked Shit is Getting Real
Assessment: SupportedSaid
Gary Marcus Observation
“Even scarier is that the employee who was tricked was suspicious, not naive, and still fell for it:”
Our reading: The deepfake scam worked even on an employee who was suspicious rather than naive.
No horizon Evidence as of
Limitation: The account of suspicion comes through police reporting, rather than a direct employee interview.
Source: Deepfaked Shit is Getting Real
Assessment: SupportedSaid
Gary Marcus Observation
“I am here to tell you that OpenAI’s research, analyzed carefully, shows no such thing, and that there is plenty of reason to be concerned.”
Our reading: OpenAI's bioweapons research, analyzed carefully, does not show that GPT-4 poses at most a slight risk of helping create biological threats, and there is plenty of reason to be concerned.
No horizon Evidence as of
Limitation: OpenAI itself acknowledged uncertainty; Marcus's criticism should not imply that the study concealed all these limitations.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: SupportedSaid
Gary Marcus Observation
“For one thing, in the expert population, the effect is in the predicted direction for all five tasks (ideation, acquisition, magnification, formulation, and release).”
Our reading: In OpenAI's threshold analysis, among experts the effect was in the predicted direction for all five tasks (ideation, acquisition, magnification, formulation and release).
No horizon Evidence as of
Limitation: These observed threshold proportions alone do not establish statistical significance or real-world biological threat capability.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: SupportedSaid
Gary Marcus Observation
“In fact, if you simply collapsed all the measurements of accuracy, and did the single most obvious test here, a simple t-test, the results would (as Footnote C implies) be significant.”
Our reading: If all the accuracy measurements in OpenAI's study were collapsed and tested with a simple t-test, the results would be statistically significant, as Footnote C implies.
No horizon Evidence as of
Limitation: The result does not establish that this analysis was uniquely appropriate or that multiple-comparison adjustment was unnecessary.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: SupportedSaid
Gary Marcus Observation
“Any fool can simply eyeball these data and see that in virtually every condition, people who have access both to the internet and LLMs (darker bars) are more accurate than people without (lighter bars).”
Our reading: In OpenAI's study, participants with both internet and LLM access were more accurate than those without in virtually every condition.
No horizon Evidence as of
Limitation: Higher observed means do not establish significance or practical danger.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: SupportedSaid
Gary Marcus Forecast
“Gary Marcus is standing firmly by his prediction that 2024 will be the year of Generative AI litigation.”
Our reading: 2024 will be the year of generative AI litigation.
Deadline passed Evidence as of
Limitation: “The year of” supplies no numerical threshold; support follows the contract's ordinary reading of substantial, sustained litigation.
Assessment: SupportedSaid
Gary Marcus Observation
“What unifies all of the above is that current systems are good at local coherence, between words, and between pixels, but not at lining up their outputs with a global comprehension of the world.”
Our reading: Current systems are good at local coherence, both between words and between pixels, but are not good at lining up their outputs with a global comprehension of the world.
No horizon Evidence as of
Limitation: Support is qualitative and depends on the stated behavioral interpretation.
Source: One GenAI image; Ten Errors
Assessment: SupportedSaid
Gary Marcus Forecast
“We can of course expect related problems of discomprehension in efforts to use these systems to address copyright issues, as I wrote earlier this week.”
Our reading: Efforts to use generative AI systems to address copyright issues will run into problems of discomprehension.
Settled before its deadline Evidence as of
Limitation: The decisive study is a preprint.
Assessment: SupportedSaid
Gary Marcus Forecast
“More than that, GenAI developers won’t get off so easy, and lawyers looking at copyright infringement will still have plenty of work to do.”
Our reading: GenAI developers will not easily escape copyright problems, and copyright infringement lawyers will have plenty of work.
Settled before its deadline Evidence as of
Limitation: The assessment treats substantial contested proceedings as the ordinary meaning of plenty of legal work.
Source: No, multimodal ChatGPT is not going to “trivially” solve Generative AI's copyright problems
Assessment: SupportedSaid
Gary Marcus Observation
“The opposite problem happens too; guardrails can wind up too permissive. This is still happening over a year into the chatbot revolution.”
Our reading: Guardrails sometimes end up too permissive, and this is still happening over a year into the chatbot revolution.
No horizon Evidence as of
Limitation: Support rests on a primary researcher summary rather than fully reported contemporary replication results.
Source: DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight
Assessment: SupportedSaid
Gary Marcus Observation
“To begin with, the threat is empty: open source LLMs are already out in the wild, and a large number of people know how to build them.”
Our reading: OpenAI's threat that generative AI would disappear is empty because open source LLMs already exist and many people can build them.
No horizon Evidence as of
Limitation: The result uses the ordinary public-model reading of open source.
Assessment: SupportedSaid
Gary Marcus Observation
“Generative AI systems like DALL-E and ChatGPT have been trained on copyrighted materials;”
Our reading: Generative AI systems such as DALL-E and ChatGPT were trained on copyrighted materials.
No horizon Evidence as of
Limitation: The records do not provide a complete inventory of training works or establish which material entered every model version.
Source: Things are about to get a lot worse for Generative AI
Assessment: SupportedSaid
Gary Marcus Observation
“It’s also hardly the first time AI is helped with mathematics.”
Our reading: FunSearch is not the first time AI has helped with mathematics.
No horizon Evidence as of
Limitation: This does not adjudicate the narrower claim that FunSearch was a first for LLM-based mathematical discovery.
Source: Sorry, but FunSearch probably isn’t a milestone in scientific discovery
Assessment: SupportedSaid
Gary Marcus Observation
“There is no evidence that the problem was heretofore “unsolvable””
Our reading: There is no evidence that the problem FunSearch addressed was previously unsolvable.
No horizon Evidence as of
Limitation: Support is limited to the ordinary distinction between impossibility and an open problem.
Source: Sorry, but FunSearch probably isn’t a milestone in scientific discovery
Assessment: SupportedSaid
Gary Marcus Observation
“Human mathematicians had to write problem-specific code for each separate mathematical problem.”
Our reading: FunSearch required human mathematicians to write problem-specific code for each separate problem.
No horizon Evidence as of
Limitation: The outcome depends on interpreting mathematicians as human task designers and separate problems as different task families.
Source: Sorry, but FunSearch probably isn’t a milestone in scientific discovery
Assessment: SupportedSaid
Gary Marcus Observation
“The whole project was quietly dropped a year or two later.”
Our reading: OpenAI quietly dropped its Rubik's cube robotics project a year or two after the demo.
No horizon Evidence as of
Limitation: The decisive company statement is reproduced in reputable news reporting; a standalone primary shutdown announcement was not retrieved.
Assessment: SupportedSaid
Gary Marcus Observation
“Note that Gates and Altman have both been dropping hints, and GPT-5 isn’t here after a year despite immense commercial desire.”
Our reading: Gates and Altman have both been dropping hints, and GPT-5 has not arrived after a year despite immense commercial desire.
No horizon Evidence as of
Limitation: Support depends on treating the year as an approximate ChatGPT-era anniversary.
Assessment: SupportedSaid
Gary Marcus Observation
“But that’s exactly what’s happening: Messrs. LeCun and Zuckerberg seem to have talked it out, and decided what’s best (in their not-so-humble opinion) for all of us.”
Our reading: Marcus says it is exactly what's happening that a couple of tech lords are deciding for us: LeCun and Zuckerberg seem to have talked out whether to open-source Meta's AI and decided what's best, in their opinion, for all of us.
No horizon Evidence as of
Limitation: Decision authority partly rests on retrospective reporting.
Source: Critical national security questions should not be decided unilaterally by unelected tech leaders
Assessment: SupportedSaid
Gary Marcus Observation
“Its pioneering, world-leading AI Act is on the ropes.”
Our reading: As of November 2023, the EU AI Act is in danger of failing.
No horizon Evidence as of
Limitation: The decisive source is reporting of participant interviews, not an official transcript or complete negotiating record.
Assessment: SupportedSaid
Gary Marcus Forecast
“And I do mean “we”; what the EU decides will affect all of us.”
Our reading: The EU's decision on regulating generative AI will affect people everywhere, not just in the EU.
Settled before its deadline Evidence as of
Limitation: Support depends on reading all of us as worldwide relevance rather than a literal claim about every person.
Assessment: SupportedSaid
Gary Marcus Observation
“getting their algorithm to work for one object (which, cheesily, turned out to be a special Cube instrumented with sensors and LEDS, unlike a Rubik’s cube you’d buy in a store) in carefully controlled lab circumstances hardly guaranteed that the solution would work more broadly in the complex open-ended real world.”
Our reading: OpenAI's 2019 Rubik's cube robot result, achieved on a sensor-instrumented cube in controlled lab conditions, did not guarantee generalization to the open-ended real world.
No horizon Evidence as of
Limitation: The selected documentation does not establish the precise claim that one demonstrated cube contained both sensors and LEDs.
Source: About that OpenAI “breakthrough”
Assessment: SupportedSaid
Gary Marcus Observation
“AV’s still aren’t general enough that you can just plop down a car that was tuned in pilot studies in Menlo Park, SF and Arizona and expect it to drive gracefully and safely in Sicily or Mumbai.”
Our reading: Autonomous vehicles still are not general enough that you can just plop down a car tuned in pilot studies in Menlo Park, SF and Arizona and expect it to drive gracefully and safely in Sicily or Mumbai.
No horizon Evidence as of
Limitation: The evidence documents Waymo's own testing process; it does not establish every developer's capabilities [e1].
Source: About that OpenAI “breakthrough”
Assessment: SupportedSaid
Gary Marcus Observation
“It’s fun to compare the two—they have so many differences—even though neither can be trusted.”
Our reading: GPT cannot be trusted.
No horizon Evidence as of
Limitation: The conclusion concerns unchecked factual reliance; the summarization results do not establish that every GPT answer is false or that GPT has no useful applications [e6].
Source: A Tale of Two Liars
Assessment: SupportedSaid
Gary Marcus Observation
“OpenAI’s strength has been in scaling up an existing paradigm (Transformers) that was developed elsewhere (Google), not developing new paradigms.”
Our reading: OpenAI's strength has been scaling the Transformer paradigm developed at Google, not developing new paradigms.
No horizon Evidence as of
Limitation: This is a qualitative assessment of the principal GPT advances, not a ranking of every OpenAI research contribution.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: SupportedSaid
Gary Marcus Observation
“And the overriding nonprofit mission was always going to be at odds with a pure for profit play.”
Our reading: OpenAI's nonprofit mission was inherently at odds with a pure for-profit business.
No horizon Evidence as of
Limitation: The outcome depends on reading at odds as a structural constraint, not perpetual operational conflict.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: SupportedSaid
Gary Marcus Observation
“There has never been a solution to the hallucination problem, and it’s not clear that there will be until we find a new paradigm, as Altman himself partly acknowledged last week, when he did his best Gary Marcus impression.”
Our reading: The hallucination problem has never been solved and may not be until a new paradigm is found.
No horizon Evidence as of
Limitation: Support concerns a general solution, not whether particular applications could achieve useful reliability.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: SupportedSaid
Gary Marcus Observation
“and then, perhaps in keeping with my surmise that Sam’s bluff was called, negotiations broke down altogether.”
Our reading: Negotiations (over Sam Altman's return to OpenAI) broke down altogether, which Marcus says is perhaps in keeping with his surmise that Altman's bluff was called.
No horizon Evidence as of
Limitation: The assessment depends on reading “altogether” within the post's immediate time frame.
Source: Everybody’s out
Assessment: SupportedSaid
Gary Marcus Observation
“We pretty obviously don’t have a say.”
Our reading: The public has no say in how OpenAI's governance crisis is resolved, including its implications for safety and humanity.
No horizon Evidence as of
Limitation: The result depends on interpreting “a say” as formal participation.
Assessment: SupportedSaid
Gary Marcus Forecast
“All signs are that those financially-interested stakeholders will quickly emerge victorious. (Arguably, they already have).”
Our reading: All signs are that OpenAI's financially interested stakeholders (employees and investors) will quickly win out over the nonprofit board, and arguably they already have.
Settled before its deadline Evidence as of
Limitation: Marcus supplied no exact deadline for 'quickly'.
Assessment: SupportedSaid
Gary Marcus Observation
“In reality, things have gone quite differently. When the nonprofit Board felt it was prudent (for undisclosed reasons) to remove Altman, per the goals of the nonprofit that they were trying to represent, at the risk of potentially devastating commercial potential, those with a stake in the nominally subordinate for-profit (both employees and investors) quickly set to work to push out the board and to undo its decisions.”
Our reading: After OpenAI's nonprofit board removed Altman, employees and investors in the for-profit quickly worked to push out the board and undo its decision, contrary to the intended structure.
No horizon Evidence as of
Limitation: News reporting is used because the private negotiations lack accessible public primary records.
Assessment: SupportedSaid
Gary Marcus Observation
“Breaking news: the draft EU AI Act—the closest thing we have to AI regulation with real teeth—is in serious jeopardy, potentially a direct victim of regulatory capture, governments doing the bidding of those they are supposed govern.”
Our reading: The draft EU AI Act is in serious jeopardy as of November 2023.
No horizon Evidence as of
Limitation: No official minutes of the 10 November meeting were obtained; the assessment relies on reporting and retrospective legal analysis.
Source: Crisis point in AI?
Assessment: SupportedSaid
Gary Marcus Observation
“Update: A few hours after I posted this Cruise CEO Kyle Vogt essentially confirmed that their “driverless” cars need very regular human intervention:”
Our reading: Cruise's CEO essentially confirmed that Cruise's driverless cars require very regular human intervention.
No horizon Evidence as of
Limitation: Supported for recurring operational assistance; strict necessity for safety is not established.
Assessment: SupportedSaid
Gary Marcus Forecast
“Shareholders will certainly sue, and if it’s bad as it looks, I doubt that GM will continue the project, which was recently suspended.”
Our reading: Shareholders will sue over Cruise.
Settled before its deadline Evidence as of
Limitation: This assessment concerns the lawsuit prediction only.
Assessment: SupportedSaid
Gary Marcus Observation
“Nobody knows for SURE that there are no serious possible consequences of open source AI, e.g in terms of bioweapons, misinformation generation etc, as Séb Krier at DeepMind nicely highlights above.”
Our reading: No one knows for certain that open source AI has no serious possible consequences, such as for bioweapons or misinformation generation.
No horizon Evidence as of
Limitation: The experiments concern public Llama 2-Chat weights and do not establish equivalent results for every open model or for access through a controlled service [e1].
Source: Where we are right now on open source and potential AI risk.
Assessment: SupportedSaid
Gary Marcus Forecast
“I—truly a techno-optimist, popular perceptions to the contrary—do think driverless will come, eventually.”
Our reading: Driverless cars will eventually arrive.
Settled before its deadline Evidence as of
Limitation: Marcus supplied no explicit adoption threshold, so interpretation materially affects the result.
Source: Why driverless cars might—or might not—be at the end of the road
Assessment: SupportedSaid
Gary Marcus Causal claim
“you don’t have to be superintelligent to create serious problems.The real issue is control.”
Our reading: AI need not be superintelligent to create serious problems; the real issue is control.
No horizon Evidence as of
Limitation: The example concerns automated driving, not an LLM.
Assessment: SupportedSaid
Gary Marcus Observation
“We have no way of guaranteeing that large language models will reliably stick to facts, a point so thoroughly established by now I don’t think it needs a reference.”
Our reading: There is no way to guarantee that large language models reliably stick to facts.
No horizon Evidence as of
Limitation: Support rests on a contemporary research synthesis, not an exhaustive proof of nonexistence.
Assessment: SupportedSaid
Gary Marcus Observation
“We have no way of guaranteeing that large language models won’t be jailbroken into saying unethical things. To the contrary, a just-published technique shows that current LLMs can be jailbroken quickly, and automatically.”
Our reading: There is no way to guarantee LLMs won't be jailbroken, and current LLMs can be jailbroken quickly and automatically.
No horizon Evidence as of
Limitation: The guarantee clause depends on the stated general-immunity interpretation.
Assessment: SupportedSaid
Gary Marcus Observation
“But, crucially, it often gets these wrong, particularly when they involve spatial reasoning beyond a cookie-cutter formula or reasoning about sequences of disparate events.”
Our reading: GPT-4 with either plug-in often gets wrong problems that a human solver would need significant thought to translate into math notation, particularly those involving spatial reasoning beyond a cookie-cutter formula or reasoning about sequences of disparate events.
No horizon Evidence as of
Limitation: The selected problems do not establish a general error rate for translation-intensive tasks or a measured comparison with human difficulty [e1].
Source: Getting GPT to work with external tools is harder than you think
Assessment: SupportedSaid
Gary Marcus Observation
“Finding a way to get LLMs to reliably use external tools is far from trivial.”
Our reading: Getting LLMs to reliably use external tools is far from trivial.
No horizon Evidence as of
Limitation: The conclusion uses an ordinary qualitative reading of difficulty.
Source: Getting GPT to work with external tools is harder than you think
Assessment: SupportedSaid
Gary Marcus Causal claim
“Certainly, adding these plug-ins does improve things.”
Our reading: Adding the Code Interpreter and Wolfram Alpha plug-ins improves GPT-4's math performance.
No horizon Evidence as of
Limitation: The shared test set comprises difficult competition problems.
Source: Getting GPT to work with external tools is harder than you think
Assessment: SupportedSaid
Gary Marcus Observation
“GPT-4 and similar current language-based AI programs notoriously struggle with mathematical calculations.”
Our reading: GPT-4 and similar current language-based AI programs struggle with mathematical calculations.
No horizon Evidence as of
Limitation: Results depend on operand sizes, operation mixtures, and prompting.
Source: Getting GPT to work with external tools is harder than you think
Assessment: SupportedSaid
Gary Marcus Observation
“On some problems that capable students might well find hard, the AI did perfectly; on other problems that lot of mathematically-inclined college students people would find easy, the AI failed miserably.”
Our reading: In the authors' tests, GPT-4 with plug-ins did perfectly on some problems that capable students might well find hard, and failed miserably on other problems that many mathematically inclined college students would find easy.
No horizon Evidence as of
Limitation: No student comparison group was tested, so the evidence does not establish student success rates or a measured ranking against students [e1].
Source: Getting GPT to work with external tools is harder than you think
Assessment: SupportedSaid
Gary Marcus Observation
“As is by now widely known, they are a hallmark of systems that generalize based on word distribution.”
Our reading: As is by now widely known, hallucinations are a hallmark of systems that generalize based on word distribution.
No horizon Evidence as of
Limitation: This establishes neither a single hallucination rate nor that all models perform equally poorly.
Source: Hello, Multimodal Hallucinations