Claims checked against evidence by Gary Marcus, page 11 of 15
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Claims checked against evidence by Gary Marcus (1471)
Each of these claims was checked against dated evidence.
Page 11 of 15: claims 1001 to 1100 of 1471, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Forecast
“This list is probably not exhaustive and will continue to grow.”
Our reading: The list of OpenAI/LLM security issues will continue to grow.
Settled before its deadline Evidence as of
Limitation: This establishes additional vulnerability disclosure, not a rate of future incidents.
Source: What Could Possibly Go Wrong with Sam Altman’s New Ambitions?
Assessment: SupportedSaid
Observation
“More evidence against his claims just keeps piling up.”
Our reading: Evidence against Hinton's claims that GenAI understands and doesn't store things keeps accumulating.
No horizon Evidence as of
Limitation: The result depends on reading the statement as an evidentiary observation rather than a conclusive theory of understanding.
Source: Further Trouble in Hinton City
Assessment: UnresolvedSaid
Observation
“GenAI systems do effectively store things. (In this case, canonical video game controller images.)”
Our reading: GenAI systems effectively store things such as canonical video game controller images.
No horizon Evidence as of
Limitation: If the controller parenthetical is treated only as an illustration, the general storage proposition is supported.
Source: Further Trouble in Hinton City
Assessment: UnresolvedSaid
Causal claim
“Noncanonical situations continue to remain difficult for generative AI systems, precisely because frequent statistics are a mediocre substitute for deep understanding.”
Our reading: Generative AI systems still struggle with noncanonical situations because frequency statistics are a poor substitute for deep understanding.
No horizon Evidence as of
Limitation: Both studies predate the target period and cover narrower image-generation settings.
Source: Further Trouble in Hinton City
Assessment: UnresolvedSaid
Observation
“They (often) don’t come up with a novel independent design that would meet the requirements.”
Our reading: GenAI systems often don't come up with a novel independent design that would meet the requirements.
No horizon Evidence as of
Limitation: The packet lacks representative evaluations jointly measuring novelty and functional compliance.
Source: Further Trouble in Hinton City
Assessment: SupportedSaid
Observation
“Reliability remain a problem.”
Our reading: Reliability remains a problem for large language models.
No horizon Evidence as of
Limitation: Benchmark performance varies by task and scoring method.
Assessment: SupportedSaid
Observation
“Scale isn’t everything, and hasn’t and won’t be enough to solve any of the problems I just listed above.”
Our reading: Scaling has not been enough to solve the problems of understanding, compositionality, negation, and reliability in LLMs.
No horizon Evidence as of
Limitation: The result depends on treating understanding as observable interpretation and generalization.
Assessment: UnresolvedSaid
Forecast
“Scale isn’t everything, and hasn’t and won’t be enough to solve any of the problems I just listed above.”
Our reading: Scaling will not be enough to solve the problems of understanding, compositionality, negation, and reliability in LLMs.
No deadline stated Evidence as of
Limitation: The boundary between scaling and a new method is unclear.
Assessment: SupportedSaid
Observation
“Compositionality remains a problem. So does negation.”
Our reading: Compositionality and negation remain problems for large language models.
No horizon Evidence as of
Limitation: Results depend on task, model version and prompt.
Assessment: UnresolvedSaid
Forecast
“But anyone who fears that GPT-5 is going to convey an immense advantage in military strategy has got to be kidding.”
Our reading: GPT-5 will not convey an immense advantage in military strategy.
No deadline stated Evidence as of
Limitation: Public evidence may omit classified deployments.
Assessment: MixedSaid
Observation
“Special-purpose AI has been great for static, easily-simulated games with rules that have been stable for centuries, like Chess, Go, and even Diplomacy, but never been great at dynamic environments in which the environment itself is unpredictable.”
Our reading: Special-purpose AI has been great for static, easily simulated games with rules that have been stable for centuries, such as chess, Go and even Diplomacy, but has never been great at dynamic environments in which the environment itself is unpredictable.
No horizon Evidence as of
Limitation: The distinction between environmental uncertainty and unrestricted task change affects the result.
Assessment: UnresolvedSaid
Observation
“Even with a lot of data, GPT-4 is playing chess at around the level where (top) chess programs were in 1976, not even close to current state of the art.”
Our reading: GPT-4 plays chess at about the level of top chess programs in 1976, far from the current state of the art.
No horizon Evidence as of
Limitation: The 1976 tournament figure comes from a secondary account [e2].
Assessment: SupportedSaid
Observation
“It’s notorious for making illegal moves in chess.”
Our reading: ChatGPT is notorious for making illegal moves in chess.
No horizon Evidence as of
Limitation: Notoriety is qualitative.
Assessment: MixedSaid
Observation
“The New York Times versus OpenAI lawsuit made that abundantly clear, with a hundred examples like this one, in which ChatGPT reconstructed entire stretches of text (red) from brief prompts2 (black text in top line):”
Our reading: The NYT v. OpenAI lawsuit showed about a hundred examples of ChatGPT reconstructing entire stretches of text from brief prompts.
No horizon Evidence as of
Limitation: Both filings are party submissions, not judicial findings about experimental validity.
Assessment: UnresolvedSaid
Causal claim
““Given vast datasets, LLMs approximate well, but their understanding is at best superficial. That’s *why* they are unreliable, and unstable, hallucinate, are constitutionally unable to fact check, etc.””
Our reading: Given vast datasets, LLMs approximate well, but their understanding is at best superficial, and that is why they are unreliable, unstable, hallucinate, are constitutionally unable to fact-check, and so on.
No horizon Evidence as of
Limitation: Partial error detection is not reliable verification of arbitrary claims.
Assessment: SupportedSaid
Observation
“But the variety of the examples along with the technical literature on memorization and data leakage in large language models make clear that this kind of reconstruction certainly can happen with some regularity.”
Our reading: Reconstruction of training text by large language models can happen with some regularity.
No horizon Evidence as of
Limitation: Marcus supplies no numerical meaning for 'some regularity'.
Assessment: SupportedSaid
Observation
“LLMs don’t store things in simple databases, but when presented with enough data, their regeneration of distributed bits has demonstrably duplicated lengthy training materials multiple times, a phenomenon that is called memorization in the field.”
Our reading: Given enough data, LLMs have demonstrably duplicated lengthy training materials multiple times (memorization).
No horizon Evidence as of
Limitation: Extraction methods can involve deliberate prompting and specialized decoding.
Assessment: UnresolvedSaid
Observation
“OpenAI will no doubt by now have filtered this specific example so that it can no longer be replicated, now that it has appeared in a lawsuit.”
Our reading: OpenAI has no doubt by now filtered this specific NYT-lawsuit example so that it can no longer be replicated, now that it has appeared in a lawsuit.
No horizon Evidence as of
Limitation: The relevant generation setup is unspecified.
Assessment: UnresolvedSaid
Causal claim
“Because of the immense amounts of data they are now trained on, and because of the immense number of parameters on which they are trained, that reconstruction can in fact sometimes come very close to memorization.”
Our reading: Because of their huge training data and parameter counts, LLM reconstruction of inputs sometimes comes very close to memorization.
No horizon Evidence as of
Limitation: This outcome concerns the full causal sentence, not whether LLMs memorize.
Assessment: UnresolvedSaid
Observation
“LLMs do (sometimes) regurgitate large chunks of effectively stored text, and they don’t understand anything at a deep level.”
Our reading: LLMs sometimes regurgitate large chunks of effectively stored text and do not understand anything at a deep level.
No horizon Evidence as of
Limitation: Closed-model training membership is inferred from matching pre-existing material rather than inspection of the complete private training corpus.
Assessment: SupportedSaid
Observation
“Neural networks do effectively memorize some of what they are trained on, and indeed there is a whole subfield now about the topic.”
Our reading: Neural networks effectively memorize some of what they are trained on, and there is now a whole subfield devoted to this topic.
No horizon Evidence as of
Limitation: For closed models, the researchers used an auxiliary corpus because the complete training data were unavailable.
Assessment: SupportedSaid
Observation
“An entire cast of deepfaked people include a deep faked CFO scammed a company out $25 million.”
Our reading: Deepfaked participants, including a deepfaked CFO, on a video call scammed a company out of $25 million.
No horizon Evidence as of
Limitation: Support depends on reading $25 million as approximate, not exact.
Source: Deepfaked Shit is Getting Real
Assessment: SupportedSaid
Observation
“Even scarier is that the employee who was tricked was suspicious, not naive, and still fell for it:”
Our reading: The deepfake scam worked even on an employee who was suspicious rather than naive.
No horizon Evidence as of
Limitation: The account of suspicion comes through police reporting, rather than a direct employee interview.
Source: Deepfaked Shit is Getting Real
Assessment: SupportedSaid
Observation
“I am here to tell you that OpenAI’s research, analyzed carefully, shows no such thing, and that there is plenty of reason to be concerned.”
Our reading: OpenAI's bioweapons research, analyzed carefully, does not show that GPT-4 poses at most a slight risk of helping create biological threats, and there is plenty of reason to be concerned.
No horizon Evidence as of
Limitation: OpenAI itself acknowledged uncertainty; Marcus's criticism should not imply that the study concealed all these limitations.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: SupportedSaid
Observation
“For one thing, in the expert population, the effect is in the predicted direction for all five tasks (ideation, acquisition, magnification, formulation, and release).”
Our reading: In OpenAI's threshold analysis, among experts the effect was in the predicted direction for all five tasks (ideation, acquisition, magnification, formulation and release).
No horizon Evidence as of
Limitation: These observed threshold proportions alone do not establish statistical significance or real-world biological threat capability.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: SupportedSaid
Observation
“In fact, if you simply collapsed all the measurements of accuracy, and did the single most obvious test here, a simple t-test, the results would (as Footnote C implies) be significant.”
Our reading: If all the accuracy measurements in OpenAI's study were collapsed and tested with a simple t-test, the results would be statistically significant, as Footnote C implies.
No horizon Evidence as of
Limitation: The result does not establish that this analysis was uniquely appropriate or that multiple-comparison adjustment was unnecessary.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: SupportedSaid
Observation
“Any fool can simply eyeball these data and see that in virtually every condition, people who have access both to the internet and LLMs (darker bars) are more accurate than people without (lighter bars).”
Our reading: In OpenAI's study, participants with both internet and LLM access were more accurate than those without in virtually every condition.
No horizon Evidence as of
Limitation: Higher observed means do not establish significance or practical danger.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: UnresolvedSaid
Observation
“At present, we do not know.”
Our reading: It is currently unknown whether open-source models without guardrails are more useful to bad actors than commercial models with guardrails.
No horizon Evidence as of
Limitation: Earlier research is context because the claim concerns the knowledge state on February 4.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: UnresolvedSaid
Observation
“It is shocking that neither OpenAI nor the media noted this (they also did not discuss in any detail one other paper that reached a somewhat similar conclusion, by Kevin Esvelt’s lab).”
Our reading: Neither OpenAI nor the media noted the fourfold difference in expert success on pathogen formulation, nor discussed in detail a similar paper from Kevin Esvelt's lab.
No horizon Evidence as of
Limitation: The supplied evidence items omit OpenAI's publication and do not provide a sufficiently comprehensive contemporary media record.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: ContradictedSaid
Observation
“As a journal reviewer, I would also note that they reported neither the p value they obtained, nor the test of statistical significance they applied; in a competent peer-reviewed journal, that’s a no-no.”
Our reading: OpenAI's paper reported neither the p value obtained nor the significance test applied, which would be unacceptable in a competent peer-reviewed journal.
No horizon Evidence as of
Limitation: The p-values are in a linked CSV rather than stated directly in footnote C.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: UnresolvedSaid
Observation
“Not every image looks like this, but it’s pretty clear after a few minutes experimentation from a friend that (a) Bard makes fantastic images but that (b) it is drawing from copyrighted sources and (c) as capable as the other systems Reid Southen and I looked at in returning derivative artwork that appears to infringe without attribution.”
Our reading: Based on a friend's few minutes of experimentation, it is pretty clear that Bard makes fantastic images, draws from copyrighted sources, and is as capable as the other systems Marcus and Reid Southen looked at in returning derivative artwork that appears to infringe without attribution, though not every image looks like this.
No horizon Evidence as of
Limitation: Plain-text sources do not provide a complete authenticated visual record.
Assessment: SupportedSaid
Forecast
“Gary Marcus is standing firmly by his prediction that 2024 will be the year of Generative AI litigation.”
Our reading: 2024 will be the year of generative AI litigation.
Deadline passed Evidence as of
Limitation: “The year of” supplies no numerical threshold; support follows the contract's ordinary reading of substantial, sustained litigation.
Assessment: SupportedSaid
Observation
“What unifies all of the above is that current systems are good at local coherence, between words, and between pixels, but not at lining up their outputs with a global comprehension of the world.”
Our reading: Current systems are good at local coherence, both between words and between pixels, but are not good at lining up their outputs with a global comprehension of the world.
No horizon Evidence as of
Limitation: Support is qualitative and depends on the stated behavioral interpretation.
Source: One GenAI image; Ten Errors
Assessment: UnresolvedSaid
Observation
“Third, at least so far OpenAI lacks both profits and a moat.”
Our reading: As of January 2024, OpenAI has neither profits nor a competitive moat.
No horizon Evidence as of
Limitation: Earlier losses cannot be carried forward as proof of current losses.
Assessment: UnresolvedSaid
Forecast
“Such errors clearly aren’t disappearing any time soon.”
Our reading: Errors of discomprehension in neural-network-based systems will not disappear any time soon.
No deadline stated Evidence as of
Limitation: Unresolved does not mean the later evidence runs against the forecast.
Assessment: UnresolvedSaid
Observation
“Examples like these, in which ChatGPT confidently generates an image that does not actually meet requirements, are nicely complement the visual hallucinations that Ernest Davis and I reported here in October, in which visual ChatGPT misinterpreted various images:”
Our reading: In examples like those shown, ChatGPT confidently generated images that did not actually meet requirements, and in examples reported in October, visual ChatGPT misinterpreted various images.
No horizon Evidence as of
Limitation: This is an access and verification gap, not evidence that Marcus's descriptions were false.
Assessment: SupportedSaid
Forecast
“We can of course expect related problems of discomprehension in efforts to use these systems to address copyright issues, as I wrote earlier this week.”
Our reading: Efforts to use generative AI systems to address copyright issues will run into problems of discomprehension.
Settled before its deadline Evidence as of
Limitation: The decisive study is a preprint.
Assessment: UnresolvedSaid
Observation
“It might be trivial to do it for the most popular characters, presented in the most canonical ways, but getting it right for all copyright materials is an entirely different matter.”
Our reading: Building a classifier that detects copyrighted material might be trivial for the most popular characters presented in the most canonical ways, but getting it right for all copyrighted materials is an entirely different matter.
No horizon Evidence as of
Limitation: The packet contains no qualifying evaluation of the January 2024 comparison.
Source: No, multimodal ChatGPT is not going to “trivially” solve Generative AI's copyright problems
Assessment: UnresolvedSaid
Observation
“Reid spent about 45 minutes looking into this earlier today (why not more? see below) with images generated by Midjourney, and it quickly became apparent that using ChatGPT as intern to trace down sources is, well, pretty intermittent.”
Our reading: In a 45-minute test, multimodal ChatGPT was unreliable at identifying the copyrighted sources of Midjourney-generated images.
No horizon Evidence as of
Limitation: The packet has no independent record of the images Southen tested or of ChatGPT's answers to them.
Source: No, multimodal ChatGPT is not going to “trivially” solve Generative AI's copyright problems
Assessment: SupportedSaid
Forecast
“More than that, GenAI developers won’t get off so easy, and lawyers looking at copyright infringement will still have plenty of work to do.”
Our reading: GenAI developers will not easily escape copyright problems, and copyright infringement lawyers will have plenty of work.
Settled before its deadline Evidence as of
Limitation: The assessment treats substantial contested proceedings as the ordinary meaning of plenty of legal work.
Source: No, multimodal ChatGPT is not going to “trivially” solve Generative AI's copyright problems
Assessment: UnresolvedSaid
Observation
“Notice, too, that Katie asked Bing “are they original?”, and Bing, untruthfully, but with a cozy emoticon, answered in the affirmative.”
Our reading: Bing falsely said its apparently infringing images were original.
No horizon Evidence as of
Limitation: The packet has no independent record of Bing's reply or of the generated images.
Source: DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight
Assessment: UnresolvedSaid
Observation
“Fundamentally, virtually any guardrail has to thread a needle between the Scylla of being too restrictive and Charybdis of being too permissive. None thus far have done this effectively.”
Our reading: No AI guardrail so far has effectively balanced being neither too restrictive nor too permissive.
No horizon Evidence as of
Limitation: The cited evaluations cover selected models, prompts, and attacks.
Source: DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight
Assessment: UnresolvedSaid
Causal claim
“Any one example can be filtered out, but there are too many examples, and current systems are too erratic to reliably get things right.”
Our reading: Because there are too many examples and current systems are too erratic, guardrails cannot reliably get things right even though any single example can be filtered.
No horizon Evidence as of
Limitation: The controlled defense study is contextual because its experiments predate the target period.
Source: DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight
Assessment: UnresolvedSaid
Observation
“Guardrail builders generally wind up playing whack-a-mole.”
Our reading: Guardrail builders generally end up patching individual failures one by one instead of solving the problem in general.
No horizon Evidence as of
Limitation: Public summaries provide less detail than internal remediation records.
Source: DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight
Assessment: UnresolvedSaid
Observation
“Other prompts still elicited apparent copyright infringements, where new filters did not suffice.”
Our reading: Even after the new filters, some prompts to Bing's DALL-E 3 still produced apparent copyright infringements.
No horizon Evidence as of
Limitation: The independent demonstration lacks authenticated generation timestamps and filter-version information.
Source: DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight
Assessment: UnresolvedSaid
Forecast
“Since that would violate the Berne Convention and infuriate every “content provider” (from individual artists and writers to major studios and publishers) ever, we hope and expect that this exemption will not be granted.”
Our reading: The UK will not grant OpenAI the exemption from copyright responsibility it requested.
No deadline stated Evidence as of
Limitation: The exemption Marcus described is not precisely identified.
Source: DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight
Assessment: SupportedSaid
Observation
“The opposite problem happens too; guardrails can wind up too permissive. This is still happening over a year into the chatbot revolution.”
Our reading: Guardrails sometimes end up too permissive, and this is still happening over a year into the chatbot revolution.
No horizon Evidence as of
Limitation: Support rests on a primary researcher summary rather than fully reported contemporary replication results.
Source: DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight
Assessment: UnresolvedSaid
Observation
“It’s likewise a flaw that that current AI cannot learn without a tendency to inadvertently regurgitate big bits of text and images that look uncannily like their sources, but not a logical necessity.”
Our reading: Current AI cannot learn without a tendency to inadvertently regurgitate big bits of text and images that look uncannily like their sources, which is a flaw but not a logical necessity.
No horizon Evidence as of
Limitation: The extraction studies do not establish an unavoidable tendency across contemporary text and image generators [e1, e2].
Assessment: UnresolvedSaid
Forecast
“LLMs may eventually be replaced by better technologies, but for now, they aren’t going anywhere, even if some of the commercial purveyors go out of business.”
Our reading: LLMs will not disappear for now even if some commercial providers go out of business, though they may eventually be replaced by better technologies.
No deadline stated Evidence as of
Limitation: Unresolved does not mean the observed trend contradicts the forecast.
Assessment: UnresolvedSaid
Observation
“OpenAI’s lobbying campaign, simply put, is based on a false dichotomy (give everything to us free or we will die)—and also a threat: either we get to use all the existing IP we want for free, or you won’t get to generative AI anymore.”
Our reading: OpenAI's copyright lobbying campaign rests on a false dichotomy and an implied threat that generative AI will go away without free use of IP.
No horizon Evidence as of
Limitation: This is an evidence-access limitation, not a finding that Marcus’s characterization is accurate.
Assessment: SupportedSaid
Observation
“To begin with, the threat is empty: open source LLMs are already out in the wild, and a large number of people know how to build them.”
Our reading: OpenAI's threat that generative AI would disappear is empty because open source LLMs already exist and many people can build them.
No horizon Evidence as of
Limitation: The result uses the ordinary public-model reading of open source.
Assessment: UnresolvedSaid
Observation
“And worse, infringement can happen even the user isn’t looking to infringe and doesn’t mention any character or film by name:”
Our reading: DALL-E can produce infringing images even when the user does not intend to infringe or name any character or film.
No horizon Evidence as of
Limitation: Reading infringement merely as recognizable copyrighted imagery would weaken the claim's stated legal commitment.
Source: Things are about to get a lot worse for Generative AI
Assessment: SupportedSaid
Observation
“Generative AI systems like DALL-E and ChatGPT have been trained on copyrighted materials;”
Our reading: Generative AI systems such as DALL-E and ChatGPT were trained on copyrighted materials.
No horizon Evidence as of
Limitation: The records do not provide a complete inventory of training works or establish which material entered every model version.
Source: Things are about to get a lot worse for Generative AI
Assessment: UnresolvedSaid
Observation
“OpenAI’s image software (which we accessed through Bing) is perfectly capable of verbatim and near-verbatim repetition of sources as well.”
Our reading: OpenAI's image generator can reproduce source images verbatim or near-verbatim.
No horizon Evidence as of
Limitation: A broader reading that counts recognizable character designs as near-verbatim sources would require different criteria.
Source: Things are about to get a lot worse for Generative AI
Assessment: UnresolvedSaid
Observation
“The output was not marked as being directly premised on copyrighted work”
Our reading: Midjourney's output was not labelled as being directly based on copyrighted work.
No horizon Evidence as of
Limitation: Absence of labelling is hard to document from secondary sources.
Source: An artist fights back, and Midjourney has embarrassed themselves
Assessment: SupportedSaid
Observation
“It’s also hardly the first time AI is helped with mathematics.”
Our reading: FunSearch is not the first time AI has helped with mathematics.
No horizon Evidence as of
Limitation: This does not adjudicate the narrower claim that FunSearch was a first for LLM-based mathematical discovery.
Source: Sorry, but FunSearch probably isn’t a milestone in scientific discovery
Assessment: SupportedSaid
Observation
“There is no evidence that the problem was heretofore “unsolvable””
Our reading: There is no evidence that the problem FunSearch addressed was previously unsolvable.
No horizon Evidence as of
Limitation: Support is limited to the ordinary distinction between impossibility and an open problem.
Source: Sorry, but FunSearch probably isn’t a milestone in scientific discovery
Assessment: SupportedSaid
Observation
“Human mathematicians had to write problem-specific code for each separate mathematical problem.”
Our reading: FunSearch required human mathematicians to write problem-specific code for each separate problem.
No horizon Evidence as of
Limitation: The outcome depends on interpreting mathematicians as human task designers and separate problems as different task families.
Source: Sorry, but FunSearch probably isn’t a milestone in scientific discovery
Assessment: SupportedSaid
Observation
“The whole project was quietly dropped a year or two later.”
Our reading: OpenAI quietly dropped its Rubik's cube robotics project a year or two after the demo.
No horizon Evidence as of
Limitation: The decisive company statement is reproduced in reputable news reporting; a standalone primary shutdown announcement was not retrieved.
Assessment: UnresolvedSaid
Forecast
“What we are going to see, in the fullness of time, I promise you, is that Gemini is more or less in the same ball park as GPT-4, handy for a bunch of things, but untethered in reality, still with dicey, unpredictable reasoning, and a very limited understanding of the world.”
Our reading: Gemini will be shown to still have dicey, unpredictable reasoning and a very limited understanding of the world.
No deadline stated Evidence as of
Limitation: Competitive benchmark performance does not prove broad world understanding.
Assessment: UnresolvedSaid
Observation
“Regular readers will recall how that turned out: the Rubik’s cube was instrumented with sensors, and not straight off the shelf, and what worked for the Rubik’s cube in the demo video didn’t turn out have a whole lot of application in the real world.”
Our reading: OpenAI's Rubik's cube robot demo relied on a sensor-instrumented cube and had little real-world application.
No horizon Evidence as of
Limitation: The evidence does not identify which demonstration video Marcus referenced.
Assessment: SupportedSaid
Observation
“Note that Gates and Altman have both been dropping hints, and GPT-5 isn’t here after a year despite immense commercial desire.”
Our reading: Gates and Altman have both been dropping hints, and GPT-5 has not arrived after a year despite immense commercial desire.
No horizon Evidence as of
Limitation: Support depends on treating the year as an approximate ChatGPT-era anniversary.
Assessment: UnresolvedSaid
Observation
“From a commercial standpoint GPT-4 is no longer unique.”
Our reading: Commercially, GPT-4 is no longer a unique offering.
No horizon Evidence as of
Limitation: The disclosures provide no matched evaluation establishing broad commercial comparability with GPT-4 on December 6 [e1, e2].
Assessment: SupportedSaid
Observation
“But that’s exactly what’s happening: Messrs. LeCun and Zuckerberg seem to have talked it out, and decided what’s best (in their not-so-humble opinion) for all of us.”
Our reading: Marcus says it is exactly what's happening that a couple of tech lords are deciding for us: LeCun and Zuckerberg seem to have talked out whether to open-source Meta's AI and decided what's best, in their opinion, for all of us.
No horizon Evidence as of
Limitation: Decision authority partly rests on retrospective reporting.
Source: Critical national security questions should not be decided unilaterally by unelected tech leaders
Assessment: UnresolvedSaid
Observation
“So far, Big Tech rather than the US government is calling the shots.”
Our reading: On sharing AI technology with other nations (e.g., OpenAI's G42 partnership), Big Tech rather than the US government is currently making the decisions.
No horizon Evidence as of
Limitation: Private diplomacy and agreement terms are incompletely public.
Source: Critical national security questions should not be decided unilaterally by unelected tech leaders
Assessment: UnresolvedSaid
Observation
“But virtually all existing laws were written before AI was really on the scene, and so very few really envision how AI changes things.”
Our reading: Virtually all existing laws were written before AI was really on the scene, and so very few of them really envision how AI changes things.
No horizon Evidence as of
Limitation: The cited evidence describes earlier enactments and is context for the publication-date legal stock.
Assessment: SupportedSaid
Observation
“Its pioneering, world-leading AI Act is on the ropes.”
Our reading: As of November 2023, the EU AI Act is in danger of failing.
No horizon Evidence as of
Limitation: The decisive source is reporting of participant interviews, not an official transcript or complete negotiating record.
Assessment: UnresolvedSaid
Observation
“Big tech wants the world to revolve around its new plaything, Generative AI (combined total revenue so far a few billion dollars, just a tiny tiny fraction of the economy), and to exempt it from any regulation.”
Our reading: Generative AI's combined total revenue so far is only a few billion dollars, a tiny fraction of the economy.
No horizon Evidence as of
Limitation: Enterprise estimates omit or differently classify consumer products and infrastructure.
Assessment: UnresolvedSaid
Observation
“Actually, there was one clear overall winner: Tegmark. As much as Ernie Davis and I utterly disagreed with Tegmark’s more technical map of the world/LLM world models paper a few weeks ago, pretty much every word he said this week about regulation was true.”
Our reading: Nearly everything Tegmark said that week about AI regulation was true.
No horizon Evidence as of
Limitation: The evaluable reading concerns factual substance; the broader endorsement also contains opinion.
Assessment: SupportedSaid
Forecast
“And I do mean “we”; what the EU decides will affect all of us.”
Our reading: The EU's decision on regulating generative AI will affect people everywhere, not just in the EU.
Settled before its deadline Evidence as of
Limitation: Support depends on reading all of us as worldwide relevance rather than a literal claim about every person.
Assessment: UnresolvedSaid
Observation
“Bing,powered by Dall-E 3, was an even bigger loser, drawing “Hinton vs LeCun vs Ng vs Tegmark, boxing” with four boxers not boxing, not one of them looking remotely like any of our heroes:”
Our reading: Bing powered by DALL-E 3, prompted to draw Hinton, LeCun, Ng and Tegmark boxing, produced four boxers not boxing and resembling none of the people named.
No horizon Evidence as of
Limitation: No visual inspection was possible.
Assessment: UnresolvedSaid
Forecast
“Eventually, we will be able to build machines that help us reason critically, overcome human bias, and objectively, the way that we have calculators that spit out just the facts, without confabulation.”
Our reading: Eventually, we will be able to build machines that help us reason critically, overcome human bias, and reason objectively, giving just the facts without confabulation, the way calculators do.
No deadline stated Evidence as of
Limitation: Small, task-specific studies cannot establish broad capability.
Source: AI versus The Clumsy Human Mind
Assessment: UnresolvedSaid
Observation
“But “breakthroughs” rarely turn to be general to live up to initial rosy expectations. Often advances work in some contexts, not otherwise.”
Our reading: Claimed AI breakthroughs rarely prove general enough to live up to initial expectations, often working only in some contexts.
No horizon Evidence as of
Limitation: The result depends on which advances and which expectations are counted.
Source: About that OpenAI “breakthrough”
Assessment: SupportedSaid
Observation
“getting their algorithm to work for one object (which, cheesily, turned out to be a special Cube instrumented with sensors and LEDS, unlike a Rubik’s cube you’d buy in a store) in carefully controlled lab circumstances hardly guaranteed that the solution would work more broadly in the complex open-ended real world.”
Our reading: OpenAI's 2019 Rubik's cube robot result, achieved on a sensor-instrumented cube in controlled lab conditions, did not guarantee generalization to the open-ended real world.
No horizon Evidence as of
Limitation: The selected documentation does not establish the precise claim that one demonstrated cube contained both sensors and LEDs.
Source: About that OpenAI “breakthrough”
Assessment: SupportedSaid
Observation
“AV’s still aren’t general enough that you can just plop down a car that was tuned in pilot studies in Menlo Park, SF and Arizona and expect it to drive gracefully and safely in Sicily or Mumbai.”
Our reading: Autonomous vehicles still are not general enough that you can just plop down a car tuned in pilot studies in Menlo Park, SF and Arizona and expect it to drive gracefully and safely in Sicily or Mumbai.
No horizon Evidence as of
Limitation: The evidence documents Waymo's own testing process; it does not establish every developer's capabilities [e1].
Source: About that OpenAI “breakthrough”
Assessment: UnresolvedSaid
Causal claim
“It autocompletes sentences, some of those completions are true given that the training data is so vast, and some aren’t, given that the training data aren’t infinite.”
Our reading: GPT autocompletes sentences; some outputs are true because training data is vast and some are false because training data is not infinite.
No horizon Evidence as of
Limitation: This assessment does not interpret finite data as the sole asserted cause.
Source: A Tale of Two Liars
Assessment: SupportedSaid
Observation
“It’s fun to compare the two—they have so many differences—even though neither can be trusted.”
Our reading: GPT cannot be trusted.
No horizon Evidence as of
Limitation: The conclusion concerns unchecked factual reliance; the summarization results do not establish that every GPT answer is false or that GPT has no useful applications [e6].
Source: A Tale of Two Liars
Assessment: UnresolvedSaid
Observation
“The real value was always the people, not the IP, not the data, not the customer list, not the infrastructure.”
Our reading: OpenAI's real value was always its people, not its IP, data, customer list or infrastructure.
No horizon Evidence as of
Limitation: Workforce and technology value may depend on each other.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: UnresolvedSaid
Causal claim
“The whole nonprofit-hybrid model, introduced in 2019 (the company was originally pure non-profit) was always a ticking time bomb, because the people trying to make money were never going to have a perfect relationship with those in the nonprofit that were charged with keeping the world safe.”
Our reading: OpenAI's nonprofit-hybrid structure was a ticking time bomb because profit-seekers and safety-focused nonprofit stewards would inevitably conflict.
No horizon Evidence as of
Limitation: Both principal causal accounts come from parties involved in the dispute.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: SupportedSaid
Observation
“OpenAI’s strength has been in scaling up an existing paradigm (Transformers) that was developed elsewhere (Google), not developing new paradigms.”
Our reading: OpenAI's strength has been scaling the Transformer paradigm developed at Google, not developing new paradigms.
No horizon Evidence as of
Limitation: This is a qualitative assessment of the principal GPT advances, not a ranking of every OpenAI research contribution.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: SupportedSaid
Observation
“And the overriding nonprofit mission was always going to be at odds with a pure for profit play.”
Our reading: OpenAI's nonprofit mission was inherently at odds with a pure for-profit business.
No horizon Evidence as of
Limitation: The outcome depends on reading at odds as a structural constraint, not perpetual operational conflict.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: SupportedSaid
Observation
“There has never been a solution to the hallucination problem, and it’s not clear that there will be until we find a new paradigm, as Altman himself partly acknowledged last week, when he did his best Gary Marcus impression.”
Our reading: The hallucination problem has never been solved and may not be until a new paradigm is found.
No horizon Evidence as of
Limitation: Support concerns a general solution, not whether particular applications could achieve useful reliability.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: UnresolvedSaid
Forecast
“Large language models alone were never going to solve the alignment problem.”
Our reading: Large language models alone were never going to solve the alignment problem.
No deadline stated Evidence as of
Limitation: The experiments do not establish reliable alignment across all relevant deployment settings.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: SupportedSaid
Observation
“and then, perhaps in keeping with my surmise that Sam’s bluff was called, negotiations broke down altogether.”
Our reading: Negotiations (over Sam Altman's return to OpenAI) broke down altogether, which Marcus says is perhaps in keeping with his surmise that Altman's bluff was called.
No horizon Evidence as of
Limitation: The assessment depends on reading “altogether” within the post's immediate time frame.
Source: Everybody’s out
Assessment: SupportedSaid
Observation
“We pretty obviously don’t have a say.”
Our reading: The public has no say in how OpenAI's governance crisis is resolved, including its implications for safety and humanity.
No horizon Evidence as of
Limitation: The result depends on interpreting “a say” as formal participation.
Assessment: MixedSaid
Observation
“The spirit of the original arrangement was that everything that the for-profit did was supposed to be in the service of the non-profit. A board with no financial interest was supposed to look out for humanity.”
Our reading: OpenAI's original structure was meant to make the for-profit serve the non-profit, with a financially disinterested board looking out for humanity.
No horizon Evidence as of
Limitation: This assesses the intended structure, not its effectiveness.
Assessment: SupportedSaid
Forecast
“All signs are that those financially-interested stakeholders will quickly emerge victorious. (Arguably, they already have).”
Our reading: All signs are that OpenAI's financially interested stakeholders (employees and investors) will quickly win out over the nonprofit board, and arguably they already have.
Settled before its deadline Evidence as of
Limitation: Marcus supplied no exact deadline for 'quickly'.
Assessment: SupportedSaid
Observation
“In reality, things have gone quite differently. When the nonprofit Board felt it was prudent (for undisclosed reasons) to remove Altman, per the goals of the nonprofit that they were trying to represent, at the risk of potentially devastating commercial potential, those with a stake in the nominally subordinate for-profit (both employees and investors) quickly set to work to push out the board and to undo its decisions.”
Our reading: After OpenAI's nonprofit board removed Altman, employees and investors in the for-profit quickly worked to push out the board and undo its decision, contrary to the intended structure.
No horizon Evidence as of
Limitation: News reporting is used because the private negotiations lack accessible public primary records.
Assessment: SupportedSaid
Observation
“Breaking news: the draft EU AI Act—the closest thing we have to AI regulation with real teeth—is in serious jeopardy, potentially a direct victim of regulatory capture, governments doing the bidding of those they are supposed govern.”
Our reading: The draft EU AI Act is in serious jeopardy as of November 2023.
No horizon Evidence as of
Limitation: No official minutes of the 10 November meeting were obtained; the assessment relies on reporting and retrospective legal analysis.
Source: Crisis point in AI?
Assessment: SupportedSaid
Observation
“Update: A few hours after I posted this Cruise CEO Kyle Vogt essentially confirmed that their “driverless” cars need very regular human intervention:”
Our reading: Cruise's CEO essentially confirmed that Cruise's driverless cars require very regular human intervention.
No horizon Evidence as of
Limitation: Supported for recurring operational assistance; strict necessity for safety is not established.
Assessment: SupportedSaid
Forecast
“Shareholders will certainly sue, and if it’s bad as it looks, I doubt that GM will continue the project, which was recently suspended.”
Our reading: Shareholders will sue over Cruise.
Settled before its deadline Evidence as of
Limitation: This assessment concerns the lawsuit prediction only.
Assessment: ContradictedSaid
Observation
“I realize there’s been almost zero public disclosure of how any of the driverless car companies are using remote operators. It’s not just Cruise.”
Our reading: Driverless car companies in general have disclosed almost nothing publicly about their use of remote operators.
No horizon Evidence as of
Limitation: The result depends on reading the claim as about disclosure of operators’ roles, rather than complete quantitative transparency.
Assessment: UnresolvedSaid
Causal claim
“Certainly no commercial product could ever work at a profit if you needed remote operators anything like that often.”
Our reading: Certainly no commercial product could ever work at a profit if it needed remote operators anything like as often as Cruise reportedly does.
No horizon Evidence as of
Only if: if remote operators were needed anything like as often as reported
Assessment: SupportedSaid
Observation
“Nobody knows for SURE that there are no serious possible consequences of open source AI, e.g in terms of bioweapons, misinformation generation etc, as Séb Krier at DeepMind nicely highlights above.”
Our reading: No one knows for certain that open source AI has no serious possible consequences, such as for bioweapons or misinformation generation.
No horizon Evidence as of
Limitation: The experiments concern public Llama 2-Chat weights and do not establish equivalent results for every open model or for access through a controlled service [e1].
Source: Where we are right now on open source and potential AI risk.
Assessment: UnresolvedSaid
Causal claim
“How effective this actually becomes depends a lot on the exact wording, and how things are enforced, and how much of it is binding as opposed to merely voluntary.”
Our reading: The Executive Order's effectiveness depends heavily on exact wording, enforcement, and how much is binding rather than voluntary.
No horizon Evidence as of
Limitation: The available evaluations cover particular activities rather than the order's overall effects.
Assessment: UnresolvedSaid
Forecast
“Companies will poke for loopholes and surely find them.”
Our reading: AI companies will look for and find loopholes in the Executive Order.
No deadline stated Evidence as of
Limitation: Company compliance deliberations may not be public.
Assessment: UnresolvedSaid
Causal claim
“To get there, though, what we need is a much smarter AI, a kind that can reason, and not hallucinate, grounded in reality and not just corpus statistics— which require fundamental research of a sort that is currently getting starved out in the LLM euphoria.”
Our reading: Getting to reliable self-driving cars that are safer than humans requires a much smarter AI that can reason, does not hallucinate, and is grounded in reality rather than just corpus statistics, and building that AI requires fundamental research.
No horizon Evidence as of
Limitation: The key capabilities and required research advances lack precise definitions.
Source: Why driverless cars might—or might not—be at the end of the road
Assessment: SupportedSaid
Forecast
“I—truly a techno-optimist, popular perceptions to the contrary—do think driverless will come, eventually.”
Our reading: Driverless cars will eventually arrive.
Settled before its deadline Evidence as of
Limitation: Marcus supplied no explicit adoption threshold, so interpretation materially affects the result.
Source: Why driverless cars might—or might not—be at the end of the road
Assessment: SupportedSaid
Causal claim
“you don’t have to be superintelligent to create serious problems.The real issue is control.”
Our reading: AI need not be superintelligent to create serious problems; the real issue is control.
No horizon Evidence as of
Limitation: The example concerns automated driving, not an LLM.
Assessment: SupportedSaid
Observation
“We have no way of guaranteeing that large language models will reliably stick to facts, a point so thoroughly established by now I don’t think it needs a reference.”
Our reading: There is no way to guarantee that large language models reliably stick to facts.
No horizon Evidence as of
Limitation: Support rests on a contemporary research synthesis, not an exhaustive proof of nonexistence.