Unresolved claims, page 6 of 8
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Unresolved claims (795)
The record was checked, and the evidence available by the cutoff does not settle it. Forecasts whose deadline has not arrived are listed apart, as not yet due.
Page 6 of 8: claims 501 to 600 of 795, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Another way to think about this (see third graph above) is that there about 5 - 7 recent models are on a par with GPT-4, but none is clearly and decisively ahead.”
Our reading: About 5 to 7 recent models are on par with GPT-4, but none is clearly ahead.
No horizon Evidence as of
Limitation: The earlier four-model list does not contradict a later approximate five-to-seven-model count.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: UnresolvedSaid
Gary Marcus Observation
“One more way to look at this, is this graph I just saw: enormous convergence on GPT-4 level performance, in multiple models released since, yet nothing decisively ahead.”
Our reading: Multiple models released since GPT-4 have converged on GPT-4-level performance with none decisively ahead.
No horizon Evidence as of
Limitation: A coding benchmark cannot establish general capability parity.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“If enthusiasm for GenAI dwindles and market valuations plummet, AI won’t disappear, and LLMs won’t disappear; they will still have their place as tools for statistical approximation.”
Our reading: If enthusiasm for GenAI dwindles and market valuations plummet, AI and LLMs will not disappear, and LLMs will still have their place as tools for statistical approximation.
No deadline stated Evidence as of
Only if: if enthusiasm for GenAI dwindles and market valuations plummet
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: UnresolvedSaid
Gary Marcus Forecast
“I couldn’t agree more. And parallel arguments could be made in many domains.”
Our reading: Marcus endorses the view that ChatGPT and similar tools will not themselves be virtual scientists anytime soon, and says parallel arguments could be made in many domains.
No deadline stated Evidence as of
Limitation: “Anytime soon” supplies no deadline; reaching the evaluation date cannot establish support.
Source: Superhuman AGI is not nigh
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“It also drives their greediness with data, and even the ethical choices their developers have been making.”
Our reading: Neural networks' trouble generalizing beyond their training space, at least in the configurations dominant over the last three decades, also drives their greediness with data and even the ethical choices their developers have been making.
No horizon Evidence as of
Limitation: The technical evidence concerns particular recurrent networks and tasks, and its link to data demand is tentative [e1].
Assessment: UnresolvedSaid
Gary Marcus Forecast
“In short, diminishing returns are what we can expect.”
Our reading: Scaling data for current AI models will yield diminishing returns.
No deadline stated Evidence as of
Limitation: A narrower claim about a specified loss curve could be supported by individual experiments.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Sooner or later, the exponential greed for data will exceed what is available.”
Our reading: The exponentially growing demand for training data will eventually exceed the data available.
No deadline stated Evidence as of
Limitation: Epoch's quantitative estimate covers public human-generated text, not all usable training data [e1].
Assessment: UnresolvedSaid
Gary Marcus Observation
“an intellectual monoculture — with no alternative approach that is nearly as well-funded.”
Our reading: The AI field is an intellectual monoculture in which no alternative approach is nearly as well-funded as generative AI.
No horizon Evidence as of
Limitation: The annual reporting period is inferred.
Assessment: UnresolvedSaid
Gary Marcus Observation
“At the same time, more optimistic predictions like these [from Reid Hoffman September 7, 2023] have not thus far borne out.”
Our reading: Reid Hoffman's optimistic September 2023 AI predictions had not come true as of March 2024.
No horizon Evidence as of
Limitation: The interview identifies the hallucination prediction but does not establish the complete set of predictions Marcus referenced. [e4]
Assessment: UnresolvedSaid
Gary Marcus Observation
“So far most uses of LLMs have been for testing, not production-level systems.”
Our reading: Most uses of LLMs so far have been for testing rather than production systems.
No horizon Evidence as of
Limitation: Survey respondents are not interchangeable with projects or usage volume.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“They are inherent to how the system makes predictions, and to the fact that LLMs themselves don’t have the wherewithal to look things up and reason over them.”
Our reading: Hallucinations are inherent to how LLMs make predictions and to their inability to look things up and reason over them.
No horizon Evidence as of
Limitation: A conditional mathematical result is not a universal result about deployed systems.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Meanwhile hype itself is becoming a kind of risk factor. The more hype, the bigger the fall, if expectations aren’t met.”
Our reading: Hype around generative AI is a risk factor: the more hype, the bigger the fall if expectations are not met.
No horizon Evidence as of
Limitation: The closest paper covers AI broadly and a particular national market.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Thus far, the closest thing to a killer app is probably coding assistace, but the income there isn’t massive enough to cover costs of the chips, legal etc.”
Our reading: Coding assistance is probably the closest thing to a generative AI killer app, but its income is not large enough to cover chip, legal and other costs.
No horizon Evidence as of
Limitation: The available financial reporting is secondary and concerns one product.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Core problems like hallucinations aren’t going way.”
Our reading: Core generative AI problems like hallucinations are not going away.
No deadline stated Evidence as of
Limitation: The cited persistence observation is from September 2025, not a complete audit of every system available at the evaluation date.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“(Not easy. With everyone building similar models, there is no moat, hence lots of competition on price).”
Our reading: Because everyone is building similar models, generative AI companies have no moat, leading to heavy price competition.
No horizon Evidence as of
Limitation: Observed convergence does not establish universal substitutability.
Assessment: UnresolvedSaid
Gary Marcus Observation
“$50B in, $3B out. That’s obviously not sustainable.”
Our reading: Generative AI investment of about $50B against $3B in revenue is obviously not sustainable.
No horizon Evidence as of
Limitation: The repeated estimate is not an independent validation of the amounts, their coverage or whether revenue was realized or annualized. [e1]
Assessment: UnresolvedSaid
Gary Marcus Observation
“Most of those who invested in driverless cars lost money, unless they invested early and cashed out, before patience began to run out.”
Our reading: Most of those who invested in driverless cars lost money, unless they invested early and cashed out before patience began to run out.
No horizon Evidence as of
Limitation: The evidence covers selected investments, not a representative population.
Source: The Second Worst $100B Investment in The History of AI?
Assessment: UnresolvedSaid
Gary Marcus Observation
“There may be a payoff someday, but thus far there has not been a lot.”
Our reading: As of March 2024, investment in driverless cars has not produced much payoff.
No horizon Evidence as of
Limitation: The company examples are not a representative industry return calculation.
Source: The Second Worst $100B Investment in The History of AI?
Assessment: UnresolvedSaid
Gary Marcus Observation
“(OpenAI even brazenly asked the British goverment to exempt them from copyright laws, wanting all the gain at none of the cost.)”
Our reading: OpenAI asked the British government to exempt it from copyright laws.
No horizon Evidence as of
Limitation: The official submission required an accessible mirror.
Source: The race between positive and negative applications of Generative AI is on – and not looking pretty
Assessment: UnresolvedSaid
Gary Marcus Observation
“Translation: But I do have equity in YCombinator and YCombinator has equity in OpenAI. It’s probably worth a lot, but you wouldn’t like me as much if I acknowledged that, so I won’t. Also, did I mention I own the OpenAI venture fund? Nope, guess not, sorry.”
Our reading: In his May 2023 Senate testimony, Sam Altman left out his indirect OpenAI stake through Y Combinator and his ownership of the OpenAI venture fund.
No horizon Evidence as of
Limitation: This is an evidentiary hold, not a finding that either interest was absent.
Source: OpenAI’s Lies and Half-Truths
Assessment: UnresolvedSaid
Gary Marcus Observation
“Two clearest examples so far this year, are the now retracted paper with the above-mentioned rat and a new lithium battery paper, with an opening sentence clearly written by an LLM.”
Our reading: In 2024 so far, a now-retracted paper with an AI-generated rat image and a lithium battery paper with an LLM-written opening sentence are the clearest examples of GenAI contaminating science.
No horizon Evidence as of
Limitation: The factual examples are strongly corroborated, but only the rat example was verified within the target period.
Source: The Year of the Big-Balled Rat
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Musk's lawsuit, combined with OpenAI's bizarre arrangement with Microsoft in which the continuation of an exclusive license rests on whether or not OpenAI has achieved something that is difficult to define (AGI) has left OpenAI in the position of needing skeptical experts to deflate their own hype.”
Our reading: Musk's lawsuit plus the AGI-contingent Microsoft license have put OpenAI in the position of needing skeptical experts to deflate its own hype.
No horizon Evidence as of
Limitation: A practical incentive is easier to establish than necessity.
Source: The Irony and the Agony of Elon Musk's Lawsuit against OpenAI
Assessment: UnresolvedSaid
Gary Marcus Observation
“I take the recital of facts in Musk’s lawsuit to be largely accurate. It is consistent with my own observations and with earlier reporting by The New York Times and The New Yorker.”
Our reading: The facts set out in Musk's lawsuit against OpenAI are largely accurate.
No horizon Evidence as of
Limitation: The original state complaint and later federal pleadings are different documents.
Source: Elon Musk sues OpenAI
Assessment: UnresolvedSaid
Gary Marcus Observation
“Elon did not get what he paid and worked for.”
Our reading: Elon Musk did not get the OpenAI he paid for and worked for.
No horizon Evidence as of
Limitation: The record does not establish Musk's acceptance of the specific structure ultimately adopted [e3].
Source: Elon Musk sues OpenAI
Assessment: UnresolvedSaid
Gary Marcus Observation
“Investors put in over $100 billion into the driverless car industry, and so far have little to show far it.”
Our reading: Investors have put over $100 billion into driverless cars and so far have little to show for it.
No horizon Evidence as of
Limitation: Deployment is not revenue, profit, or investor return.
Source: No, RAG is probably not going to rescue the current situation
Assessment: UnresolvedSaid
Gary Marcus Forecast
“RAG may reduce hallucinations somewhat, but it certainly won’t come close to eliminating them.”
Our reading: RAG may somewhat reduce hallucinations but will not come close to eliminating them.
No deadline stated Evidence as of
Limitation: The claim supplies no deadline or numerical definition of near-elimination.
Source: No, RAG is probably not going to rescue the current situation
Assessment: UnresolvedSaid
Gary Marcus Observation
“There’s no reason to think that any such thing yet exists, nor that building such a comparison tool is any easier than solving AGI itself.”
Our reading: There is no reason to think that a technology yet exists for comparing an LLM's output with the text of a reliable source and determining with very high accuracy whether they say the same thing, nor that building such a comparison tool is any easier than solving AGI.
No horizon Evidence as of
Limitation: The compound claim combines an empirical existence question with an undefined difficulty comparison.
Source: No, RAG is probably not going to rescue the current situation
Assessment: UnresolvedSaid
Gary Marcus Observation
“Gary Marcus has said from day 1 that LLMs are oversold.”
Our reading: Gary Marcus has said from day 1 that LLMs are oversold.
No horizon Evidence as of
Limitation: The documented criticism concerns GPT-2 and cannot establish his entire earlier record.
Source: Belief in magic may be declining
Assessment: UnresolvedSaid
Gary Marcus Observation
“The reality, though is that these systems have never been been stable.”
Our reading: Generative AI systems have never been stable.
No horizon Evidence as of
Limitation: Stable has an ordinary meaning, but its boundary is not specified.
Source: ChatGPT has gone berserk
Assessment: UnresolvedSaid
Gary Marcus Observation
“Current systems mix historical data, images from old films etc, all together and are way too unsophisticated to determine what would count as representative current sample.”
Our reading: Current AI systems mix historical and current data indiscriminately and are too unsophisticated to determine a representative current sample.
No horizon Evidence as of
Limitation: The demographic experiments predate the target period.
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: UnresolvedSaid
Gary Marcus Observation
“But here’s the thing. The AI we have got now isn’t really up to the job. Getting the delicate balance here right is way outside current AI’s wheelhouse.”
Our reading: Current AI is not capable of getting the balance between cultural sensitivity and historical accuracy right.
No horizon Evidence as of
Limitation: The result depends on retaining the claim's general scope rather than narrowing it to Gemini.
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Getting AI to behave in a culturally-sensitive yet historically accurate way is hard for two reasons, one having to do with data and cultural change, the other with intelligence.”
Our reading: Making AI culturally sensitive yet historically accurate is hard because of skewed data/cultural change and AI's lack of intelligence.
No horizon Evidence as of
Limitation: Google's explanation is a company account rather than an independently reproduced causal investigation.
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: UnresolvedSaid
Gary Marcus Observation
“It should try to clarify if it was unsure of the request. But current AI just ain’t that smart.”
Our reading: Current AI is not smart enough to behave as a sensible system should, for example by telling historical requests from contemporary ones and requests for real photos from requests for fictional characters, and by asking for clarification when unsure of a request.
No horizon Evidence as of
Limitation: Neither item supplies an evaluation matched to the February 2024 deployments and the full set of request distinctions [e6, e7].
Source: Has Google gone too woke? Why even the biggest models still struggle with guardrails
Assessment: UnresolvedSaid
Gary Marcus Forecast
“We are about to have a whole generation of children educated by fake videos that are completely plausible to naive audiences yet biologically (or physically or culturally, etc) incorrect.”
Our reading: Soon, a whole generation of children will be educated by AI-generated fake videos that look plausible to naive viewers but are factually incorrect.
No deadline stated Evidence as of
Limitation: The scope words ('whole generation', 'educated by') are hard to measure.
Source: Absolutely amazing ant video augurs intellectual mayhem
Assessment: UnresolvedSaid
Gary Marcus Observation
“But it’s drawing the wrong generalization.”
Our reading: Sora arguably shows some generalization, but in the monkey-chess case it is drawing the wrong generalization.
No horizon Evidence as of
Limitation: The evidence packet has no independent description or analysis of the original monkey-chess video.
Source: Sora Can’t Handle the Truth
Assessment: UnresolvedSaid
Gary Marcus Observation
“Sora fundamentally revolves around trying to take pixels and reconstruct what pixels might be nearby in space and time. The forest is never perceived apart from the trees, and the chessboard is never perceived apart from its constituent squares.”
Our reading: Sora fundamentally works by reconstructing nearby pixels in space and time rather than perceiving whole objects.
No horizon Evidence as of
Limitation: The cited review appeared after the post and is used only as context about the original model.
Source: Sora Can’t Handle the Truth
Assessment: UnresolvedSaid
Gary Marcus Forecast
“More data won’t solve that problem.”
Our reading: More training data will not solve Sora's hallucination-like physics glitches.
No deadline stated Evidence as of
Limitation: The research experiments test models other than OpenAI's Sora and do not establish what all larger datasets could achieve [e1, e2].
Source: Sora’s Surreal Physics
Assessment: UnresolvedSaid
Gary Marcus Forecast
“And importantly, I predict that many will be hard to remedy.”
Our reading: Many of Sora's physics glitches will be hard to fix.
No deadline stated Evidence as of
Limitation: Public development disclosures provide little detail about targeted repair effort.
Source: Sora’s Surreal Physics
Assessment: UnresolvedSaid
Gary Marcus Observation
“If the system really is being “programmed in English” it is not always doing what is being asked to do:”
Our reading: If ChatGPT/GPT-4 really is being "programmed in English" through its allegedly leaked system prompt, it is not always doing what it is asked to do.
No horizon Evidence as of
Limitation: An absent authentication record does not show that the alleged prompt was fabricated.
Source: The hottest new programming language is English! Or maybe not.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Much of what Reid Southen, Katie Conrad, and I recently described is in flagrant violation of those directives.”
Our reading: Much of what Marcus, Reid Southen and Katie Conrad recently described is in flagrant violation of the directives in ChatGPT's allegedly leaked system prompt.
No horizon Evidence as of
Limitation: This assessment concerns compliance with instructions, not whether an image legally infringes copyright.
Source: The hottest new programming language is English! Or maybe not.
Assessment: UnresolvedSaid
Gary Marcus Observation
“One winner out of three to be exact.”
Our reading: After several retries, Gemini produced only one correct horse-riding-astronaut image out of three.
No horizon Evidence as of
Limitation: This concerns one historical response, not Gemini's general success rate.
Source: Horse rides Astronaut, redux
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“In other words, the problem was with its language understanding, rather than with illustration per se. The model was perfectly capable of drawing a horse riding an astronaut, it just didn’t know that the phrase horse rides astronaut was a request for a horse riding an astronaut.”
Our reading: DALL-E's failure on 'horse rides astronaut' was due to deficient language understanding, not illustration ability.
No horizon Evidence as of
Limitation: The original images and complete sampling records were not available in the retrieved text.
Source: Horse rides Astronaut, redux
Assessment: UnresolvedSaid
Gary Marcus Observation
“I think we can safely reach the same conclusion as before. Gemini can draw the desired image. But it’s still like pulling teeth to get there.”
Our reading: Gemini can draw a horse riding an astronaut, but getting it to do so is very difficult.
No horizon Evidence as of
Limitation: A verified difficult session would support the local observation more strongly than a general claim about typical performance.
Source: Horse rides Astronaut, redux
Assessment: UnresolvedSaid
Gary Marcus Observation
“In our own brief experiments with GPT-3 (OpenAI has refused us proper scientific access for over a year) we found cases like the following, which reflects a complete failure to understand human biology.”
Our reading: In Marcus and Davis's experiments, GPT-3 produced outputs showing a complete failure to understand human biology.
No horizon Evidence as of
Limitation: The local-error reading is narrower than a claim that GPT-3 understood nothing about biology.
Source: Statistics versus Understanding: The Essence of What Ails Generative AI
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“That’s why statistically improbable requests like horses riding astronauts have always been challenging for generative image systems.”
Our reading: Because they rely on statistics instead of understanding, generative image systems have always struggled with statistically improbable requests such as horses riding astronauts.
No horizon Evidence as of
Limitation: The most directly relevant study concerns one model.
Source: Statistics versus Understanding: The Essence of What Ails Generative AI
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Once again statistical frequency (most guitarists play righthanded) won out over linguistic understanding, to the consternation of Hendrix fans everywhere.”
Our reading: In Wyatt Walls's left-handed guitarist example, statistical frequency (most guitarists play right-handed) once again won out over linguistic understanding.
No horizon Evidence as of
Limitation: Image-based tweets could not be inspected.
Source: Statistics versus Understanding: The Essence of What Ails Generative AI
Assessment: UnresolvedSaid
Gary Marcus Observation
“Negative externalities, essentially none of which OpenAI is absorbing, like the cost of dis- and misinformation, cybercrimes, and the latest, fake books, a scam that has recently hurt at least three authors I know over the last two months.”
Our reading: OpenAI absorbs essentially none of the negative externalities of its products, such as misinformation, cybercrime, and fake books.
No horizon Evidence as of
Limitation: Private payments and remediation costs may not be publicly disclosed.
Source: Seven reasons why the world should say No to Sam Altman
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“They are already a target for attack, and as long as they are successful they will continue to be; if they have direct access to user devices, the risks of them getting hacked will increase, and the spoils of war for the attackers will increase, since it enables essentially infinite mischief (e.g., draining bank balances, stealing credit card numbers, placing fraudulent transactions, extortion based on private photos, holding devices for ransom, and so on).”
Our reading: If OpenAI gains direct access to user devices, the risk of it being hacked and the rewards to attackers will increase.
No deadline stated Evidence as of
Only if: if OpenAI has direct access to user devices
Source: What Could Possibly Go Wrong with Sam Altman’s New Ambitions?
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Chatbots are struggling to make the money people envision, so he is doubling down to (a) agents, and (b) a multitrillion-dollar chips-and-energy infrastructure play.”
Our reading: Chatbots are struggling to make the money people envision, so Altman is doubling down on agents and on a multitrillion-dollar chips-and-energy infrastructure play.
No horizon Evidence as of
Limitation: The financial measure and expected result are unspecified.
Source: What Could Possibly Go Wrong with Sam Altman’s New Ambitions?
Assessment: UnresolvedSaid
Gary Marcus Observation
“GenAI systems do effectively store things. (In this case, canonical video game controller images.)”
Our reading: GenAI systems effectively store things such as canonical video game controller images.
No horizon Evidence as of
Limitation: If the controller parenthetical is treated only as an illustration, the general storage proposition is supported.
Source: Further Trouble in Hinton City
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Noncanonical situations continue to remain difficult for generative AI systems, precisely because frequent statistics are a mediocre substitute for deep understanding.”
Our reading: Generative AI systems still struggle with noncanonical situations because frequency statistics are a poor substitute for deep understanding.
No horizon Evidence as of
Limitation: Both studies predate the target period and cover narrower image-generation settings.
Source: Further Trouble in Hinton City
Assessment: UnresolvedSaid
Gary Marcus Observation
“They (often) don’t come up with a novel independent design that would meet the requirements.”
Our reading: GenAI systems often don't come up with a novel independent design that would meet the requirements.
No horizon Evidence as of
Limitation: The packet lacks representative evaluations jointly measuring novelty and functional compliance.
Source: Further Trouble in Hinton City
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Scale isn’t everything, and hasn’t and won’t be enough to solve any of the problems I just listed above.”
Our reading: Scaling will not be enough to solve the problems of understanding, compositionality, negation, and reliability in LLMs.
No deadline stated Evidence as of
Limitation: The boundary between scaling and a new method is unclear.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“But anyone who fears that GPT-5 is going to convey an immense advantage in military strategy has got to be kidding.”
Our reading: GPT-5 will not convey an immense advantage in military strategy.
No deadline stated Evidence as of
Limitation: Public evidence may omit classified deployments.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Even with a lot of data, GPT-4 is playing chess at around the level where (top) chess programs were in 1976, not even close to current state of the art.”
Our reading: GPT-4 plays chess at about the level of top chess programs in 1976, far from the current state of the art.
No horizon Evidence as of
Limitation: The 1976 tournament figure comes from a secondary account [e2].
Assessment: UnresolvedSaid
Gary Marcus Causal claim
““Given vast datasets, LLMs approximate well, but their understanding is at best superficial. That’s *why* they are unreliable, and unstable, hallucinate, are constitutionally unable to fact check, etc.””
Our reading: Given vast datasets, LLMs approximate well, but their understanding is at best superficial, and that is why they are unreliable, unstable, hallucinate, are constitutionally unable to fact-check, and so on.
No horizon Evidence as of
Limitation: Partial error detection is not reliable verification of arbitrary claims.
Assessment: UnresolvedSaid
Gary Marcus Observation
“OpenAI will no doubt by now have filtered this specific example so that it can no longer be replicated, now that it has appeared in a lawsuit.”
Our reading: OpenAI has no doubt by now filtered this specific NYT-lawsuit example so that it can no longer be replicated, now that it has appeared in a lawsuit.
No horizon Evidence as of
Limitation: The relevant generation setup is unspecified.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Because of the immense amounts of data they are now trained on, and because of the immense number of parameters on which they are trained, that reconstruction can in fact sometimes come very close to memorization.”
Our reading: Because of their huge training data and parameter counts, LLM reconstruction of inputs sometimes comes very close to memorization.
No horizon Evidence as of
Limitation: This outcome concerns the full causal sentence, not whether LLMs memorize.
Assessment: UnresolvedSaid
Gary Marcus Observation
“LLMs do (sometimes) regurgitate large chunks of effectively stored text, and they don’t understand anything at a deep level.”
Our reading: LLMs sometimes regurgitate large chunks of effectively stored text and do not understand anything at a deep level.
No horizon Evidence as of
Limitation: Closed-model training membership is inferred from matching pre-existing material rather than inspection of the complete private training corpus.
Assessment: UnresolvedSaid
Gary Marcus Observation
“At present, we do not know.”
Our reading: It is currently unknown whether open-source models without guardrails are more useful to bad actors than commercial models with guardrails.
No horizon Evidence as of
Limitation: Earlier research is context because the claim concerns the knowledge state on February 4.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: UnresolvedSaid
Gary Marcus Observation
“It is shocking that neither OpenAI nor the media noted this (they also did not discuss in any detail one other paper that reached a somewhat similar conclusion, by Kevin Esvelt’s lab).”
Our reading: Neither OpenAI nor the media noted the fourfold difference in expert success on pathogen formulation, nor discussed in detail a similar paper from Kevin Esvelt's lab.
No horizon Evidence as of
Limitation: The supplied evidence items omit OpenAI's publication and do not provide a sufficiently comprehensive contemporary media record.
Source: Diving deep into OpenAI’s new study on LLM’s and bioweapons
Assessment: UnresolvedSaid
Gary Marcus Observation
“Not every image looks like this, but it’s pretty clear after a few minutes experimentation from a friend that (a) Bard makes fantastic images but that (b) it is drawing from copyrighted sources and (c) as capable as the other systems Reid Southen and I looked at in returning derivative artwork that appears to infringe without attribution.”
Our reading: Based on a friend's few minutes of experimentation, it is pretty clear that Bard makes fantastic images, draws from copyrighted sources, and is as capable as the other systems Marcus and Reid Southen looked at in returning derivative artwork that appears to infringe without attribution, though not every image looks like this.
No horizon Evidence as of
Limitation: Plain-text sources do not provide a complete authenticated visual record.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Third, at least so far OpenAI lacks both profits and a moat.”
Our reading: As of January 2024, OpenAI has neither profits nor a competitive moat.
No horizon Evidence as of
Limitation: Earlier losses cannot be carried forward as proof of current losses.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Such errors clearly aren’t disappearing any time soon.”
Our reading: Errors of discomprehension in neural-network-based systems will not disappear any time soon.
No deadline stated Evidence as of
Limitation: Unresolved does not mean the later evidence runs against the forecast.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Examples like these, in which ChatGPT confidently generates an image that does not actually meet requirements, are nicely complement the visual hallucinations that Ernest Davis and I reported here in October, in which visual ChatGPT misinterpreted various images:”
Our reading: In examples like those shown, ChatGPT confidently generated images that did not actually meet requirements, and in examples reported in October, visual ChatGPT misinterpreted various images.
No horizon Evidence as of
Limitation: This is an access and verification gap, not evidence that Marcus's descriptions were false.
Assessment: UnresolvedSaid
Gary Marcus Observation
“It might be trivial to do it for the most popular characters, presented in the most canonical ways, but getting it right for all copyright materials is an entirely different matter.”
Our reading: Building a classifier that detects copyrighted material might be trivial for the most popular characters presented in the most canonical ways, but getting it right for all copyrighted materials is an entirely different matter.
No horizon Evidence as of
Limitation: The packet contains no qualifying evaluation of the January 2024 comparison.
Source: No, multimodal ChatGPT is not going to “trivially” solve Generative AI's copyright problems
Assessment: UnresolvedSaid
Gary Marcus Observation
“Reid spent about 45 minutes looking into this earlier today (why not more? see below) with images generated by Midjourney, and it quickly became apparent that using ChatGPT as intern to trace down sources is, well, pretty intermittent.”
Our reading: In a 45-minute test, multimodal ChatGPT was unreliable at identifying the copyrighted sources of Midjourney-generated images.
No horizon Evidence as of
Limitation: The packet has no independent record of the images Southen tested or of ChatGPT's answers to them.
Source: No, multimodal ChatGPT is not going to “trivially” solve Generative AI's copyright problems
Assessment: UnresolvedSaid
Gary Marcus Observation
“Notice, too, that Katie asked Bing “are they original?”, and Bing, untruthfully, but with a cozy emoticon, answered in the affirmative.”
Our reading: Bing falsely said its apparently infringing images were original.
No horizon Evidence as of
Limitation: The packet has no independent record of Bing's reply or of the generated images.
Source: DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight
Assessment: UnresolvedSaid
Gary Marcus Observation
“Fundamentally, virtually any guardrail has to thread a needle between the Scylla of being too restrictive and Charybdis of being too permissive. None thus far have done this effectively.”
Our reading: No AI guardrail so far has effectively balanced being neither too restrictive nor too permissive.
No horizon Evidence as of
Limitation: The cited evaluations cover selected models, prompts, and attacks.
Source: DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Any one example can be filtered out, but there are too many examples, and current systems are too erratic to reliably get things right.”
Our reading: Because there are too many examples and current systems are too erratic, guardrails cannot reliably get things right even though any single example can be filtered.
No horizon Evidence as of
Limitation: The controlled defense study is contextual because its experiments predate the target period.
Source: DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight
Assessment: UnresolvedSaid
Gary Marcus Observation
“Guardrail builders generally wind up playing whack-a-mole.”
Our reading: Guardrail builders generally end up patching individual failures one by one instead of solving the problem in general.
No horizon Evidence as of
Limitation: Public summaries provide less detail than internal remediation records.
Source: DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight
Assessment: UnresolvedSaid
Gary Marcus Observation
“Other prompts still elicited apparent copyright infringements, where new filters did not suffice.”
Our reading: Even after the new filters, some prompts to Bing's DALL-E 3 still produced apparent copyright infringements.
No horizon Evidence as of
Limitation: The independent demonstration lacks authenticated generation timestamps and filter-version information.
Source: DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Since that would violate the Berne Convention and infuriate every “content provider” (from individual artists and writers to major studios and publishers) ever, we hope and expect that this exemption will not be granted.”
Our reading: The UK will not grant OpenAI the exemption from copyright responsibility it requested.
No deadline stated Evidence as of
Limitation: The exemption Marcus described is not precisely identified.
Source: DALL-E’s New Guardrails: Fast, Furious, and Far from Airtight
Assessment: UnresolvedSaid
Gary Marcus Observation
“It’s likewise a flaw that that current AI cannot learn without a tendency to inadvertently regurgitate big bits of text and images that look uncannily like their sources, but not a logical necessity.”
Our reading: Current AI cannot learn without a tendency to inadvertently regurgitate big bits of text and images that look uncannily like their sources, which is a flaw but not a logical necessity.
No horizon Evidence as of
Limitation: The extraction studies do not establish an unavoidable tendency across contemporary text and image generators [e1, e2].
Assessment: UnresolvedSaid
Gary Marcus Forecast
“LLMs may eventually be replaced by better technologies, but for now, they aren’t going anywhere, even if some of the commercial purveyors go out of business.”
Our reading: LLMs will not disappear for now even if some commercial providers go out of business, though they may eventually be replaced by better technologies.
No deadline stated Evidence as of
Limitation: Unresolved does not mean the observed trend contradicts the forecast.
Assessment: UnresolvedSaid
Gary Marcus Observation
“OpenAI’s lobbying campaign, simply put, is based on a false dichotomy (give everything to us free or we will die)—and also a threat: either we get to use all the existing IP we want for free, or you won’t get to generative AI anymore.”
Our reading: OpenAI's copyright lobbying campaign rests on a false dichotomy and an implied threat that generative AI will go away without free use of IP.
No horizon Evidence as of
Limitation: This is an evidence-access limitation, not a finding that Marcus’s characterization is accurate.
Assessment: UnresolvedSaid
Gary Marcus Observation
“And worse, infringement can happen even the user isn’t looking to infringe and doesn’t mention any character or film by name:”
Our reading: DALL-E can produce infringing images even when the user does not intend to infringe or name any character or film.
No horizon Evidence as of
Limitation: Reading infringement merely as recognizable copyrighted imagery would weaken the claim's stated legal commitment.
Source: Things are about to get a lot worse for Generative AI
Assessment: UnresolvedSaid
Gary Marcus Observation
“OpenAI’s image software (which we accessed through Bing) is perfectly capable of verbatim and near-verbatim repetition of sources as well.”
Our reading: OpenAI's image generator can reproduce source images verbatim or near-verbatim.
No horizon Evidence as of
Limitation: A broader reading that counts recognizable character designs as near-verbatim sources would require different criteria.
Source: Things are about to get a lot worse for Generative AI
Assessment: UnresolvedSaid
Gary Marcus Observation
“The output was not marked as being directly premised on copyrighted work”
Our reading: Midjourney's output was not labelled as being directly based on copyrighted work.
No horizon Evidence as of
Limitation: Absence of labelling is hard to document from secondary sources.
Source: An artist fights back, and Midjourney has embarrassed themselves
Assessment: UnresolvedSaid
Gary Marcus Forecast
“What we are going to see, in the fullness of time, I promise you, is that Gemini is more or less in the same ball park as GPT-4, handy for a bunch of things, but untethered in reality, still with dicey, unpredictable reasoning, and a very limited understanding of the world.”
Our reading: Gemini will be shown to still have dicey, unpredictable reasoning and a very limited understanding of the world.
No deadline stated Evidence as of
Limitation: Competitive benchmark performance does not prove broad world understanding.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Regular readers will recall how that turned out: the Rubik’s cube was instrumented with sensors, and not straight off the shelf, and what worked for the Rubik’s cube in the demo video didn’t turn out have a whole lot of application in the real world.”
Our reading: OpenAI's Rubik's cube robot demo relied on a sensor-instrumented cube and had little real-world application.
No horizon Evidence as of
Limitation: The evidence does not identify which demonstration video Marcus referenced.
Assessment: UnresolvedSaid
Gary Marcus Observation
“From a commercial standpoint GPT-4 is no longer unique.”
Our reading: Commercially, GPT-4 is no longer a unique offering.
No horizon Evidence as of
Limitation: The disclosures provide no matched evaluation establishing broad commercial comparability with GPT-4 on December 6 [e1, e2].
Assessment: UnresolvedSaid
Gary Marcus Observation
“So far, Big Tech rather than the US government is calling the shots.”
Our reading: On sharing AI technology with other nations (e.g., OpenAI's G42 partnership), Big Tech rather than the US government is currently making the decisions.
No horizon Evidence as of
Limitation: Private diplomacy and agreement terms are incompletely public.
Source: Critical national security questions should not be decided unilaterally by unelected tech leaders
Assessment: UnresolvedSaid
Gary Marcus Observation
“But virtually all existing laws were written before AI was really on the scene, and so very few really envision how AI changes things.”
Our reading: Virtually all existing laws were written before AI was really on the scene, and so very few of them really envision how AI changes things.
No horizon Evidence as of
Limitation: The cited evidence describes earlier enactments and is context for the publication-date legal stock.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Big tech wants the world to revolve around its new plaything, Generative AI (combined total revenue so far a few billion dollars, just a tiny tiny fraction of the economy), and to exempt it from any regulation.”
Our reading: Generative AI's combined total revenue so far is only a few billion dollars, a tiny fraction of the economy.
No horizon Evidence as of
Limitation: Enterprise estimates omit or differently classify consumer products and infrastructure.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Actually, there was one clear overall winner: Tegmark. As much as Ernie Davis and I utterly disagreed with Tegmark’s more technical map of the world/LLM world models paper a few weeks ago, pretty much every word he said this week about regulation was true.”
Our reading: Nearly everything Tegmark said that week about AI regulation was true.
No horizon Evidence as of
Limitation: The evaluable reading concerns factual substance; the broader endorsement also contains opinion.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Bing,powered by Dall-E 3, was an even bigger loser, drawing “Hinton vs LeCun vs Ng vs Tegmark, boxing” with four boxers not boxing, not one of them looking remotely like any of our heroes:”
Our reading: Bing powered by DALL-E 3, prompted to draw Hinton, LeCun, Ng and Tegmark boxing, produced four boxers not boxing and resembling none of the people named.
No horizon Evidence as of
Limitation: No visual inspection was possible.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Eventually, we will be able to build machines that help us reason critically, overcome human bias, and objectively, the way that we have calculators that spit out just the facts, without confabulation.”
Our reading: Eventually, we will be able to build machines that help us reason critically, overcome human bias, and reason objectively, giving just the facts without confabulation, the way calculators do.
No deadline stated Evidence as of
Limitation: Small, task-specific studies cannot establish broad capability.
Source: AI versus The Clumsy Human Mind
Assessment: UnresolvedSaid
Gary Marcus Observation
“But “breakthroughs” rarely turn to be general to live up to initial rosy expectations. Often advances work in some contexts, not otherwise.”
Our reading: Claimed AI breakthroughs rarely prove general enough to live up to initial expectations, often working only in some contexts.
No horizon Evidence as of
Limitation: The result depends on which advances and which expectations are counted.
Source: About that OpenAI “breakthrough”
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“It autocompletes sentences, some of those completions are true given that the training data is so vast, and some aren’t, given that the training data aren’t infinite.”
Our reading: GPT autocompletes sentences; some outputs are true because training data is vast and some are false because training data is not infinite.
No horizon Evidence as of
Limitation: This assessment does not interpret finite data as the sole asserted cause.
Source: A Tale of Two Liars
Assessment: UnresolvedSaid
Gary Marcus Observation
“The real value was always the people, not the IP, not the data, not the customer list, not the infrastructure.”
Our reading: OpenAI's real value was always its people, not its IP, data, customer list or infrastructure.
No horizon Evidence as of
Limitation: Workforce and technology value may depend on each other.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“The whole nonprofit-hybrid model, introduced in 2019 (the company was originally pure non-profit) was always a ticking time bomb, because the people trying to make money were never going to have a perfect relationship with those in the nonprofit that were charged with keeping the world safe.”
Our reading: OpenAI's nonprofit-hybrid structure was a ticking time bomb because profit-seekers and safety-focused nonprofit stewards would inevitably conflict.
No horizon Evidence as of
Limitation: Both principal causal accounts come from parties involved in the dispute.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Large language models alone were never going to solve the alignment problem.”
Our reading: Large language models alone were never going to solve the alignment problem.
No deadline stated Evidence as of
Limitation: The experiments do not establish reliable alignment across all relevant deployment settings.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Certainly no commercial product could ever work at a profit if you needed remote operators anything like that often.”
Our reading: Certainly no commercial product could ever work at a profit if it needed remote operators anything like as often as Cruise reportedly does.
No horizon Evidence as of
Only if: if remote operators were needed anything like as often as reported
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“How effective this actually becomes depends a lot on the exact wording, and how things are enforced, and how much of it is binding as opposed to merely voluntary.”
Our reading: The Executive Order's effectiveness depends heavily on exact wording, enforcement, and how much is binding rather than voluntary.
No horizon Evidence as of
Limitation: The available evaluations cover particular activities rather than the order's overall effects.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Companies will poke for loopholes and surely find them.”
Our reading: AI companies will look for and find loopholes in the Executive Order.
No deadline stated Evidence as of
Limitation: Company compliance deliberations may not be public.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“To get there, though, what we need is a much smarter AI, a kind that can reason, and not hallucinate, grounded in reality and not just corpus statistics— which require fundamental research of a sort that is currently getting starved out in the LLM euphoria.”
Our reading: Getting to reliable self-driving cars that are safer than humans requires a much smarter AI that can reason, does not hallucinate, and is grounded in reality rather than just corpus statistics, and building that AI requires fundamental research.
No horizon Evidence as of
Limitation: The key capabilities and required research advances lack precise definitions.
Source: Why driverless cars might—or might not—be at the end of the road
Assessment: UnresolvedSaid
Gary Marcus Observation
“Current AI has no way to reliably anticipate consequences of actions that it might recommend, Asimov’s First Law of Robotics called for robots to do no harm, but current AI has no reliable way to anticipate harm.”
Our reading: Current AI cannot reliably anticipate the consequences or harms of actions it recommends.
No horizon Evidence as of
Limitation: The scope of current AI and the meaning of reliably are unspecified.
Assessment: UnresolvedSaid
Gary Marcus Observation
“In a few cases GPT-4 set up the calculations correctly and the plug-in returned the correct answer, but then GPT-4 either misunderstood or refused to accept that answer.”
Our reading: In a few test cases GPT-4 set up the calculations correctly and the plug-in returned the correct answer, but GPT-4 then either misunderstood or refused to accept that answer.
No horizon Evidence as of
Limitation: A 2024 transcript revision supplies important qualifications about the 2023 tests.
Source: Getting GPT to work with external tools is harder than you think
Assessment: UnresolvedSaid
Gary Marcus Observation
“Thus far we have seen consistently unreliable results, from math to fact-checking.”
Our reading: So far, the authors have seen consistently unreliable results from LLMs using external tools, from math to fact-checking.
No horizon Evidence as of
Limitation: The outcome depends on retaining the explicit personal scope, 'we have seen'.
Source: Getting GPT to work with external tools is harder than you think