Unresolved claims, page 8 of 8
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Unresolved claims (795)
The record was checked, and the evidence available by the cutoff does not settle it. Forecasts whose deadline has not arrived are listed apart, as not yet due.
Page 8 of 8: claims 701 to 795 of 795, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: UnresolvedSaid
Gary Marcus Observation
“But more than three years on, it’s still—even after a new update a few days ago focused around math—having lot of problems with easy word problems like this:”
Our reading: Even after its January 30, 2023 math-focused update, ChatGPT still has many problems with easy word problems.
No horizon Evidence as of
Limitation: Missing image contents prevent verification of the examples Marcus cited.
Source: Gaslighting and reality in AI
Assessment: UnresolvedSaid
Gary Marcus Observation
“I said the problem wasn’t math per se, but that large language models didn’t represent robust world models of how events fold over time; I stand by this 100% even today.”
Our reading: Large language models do not represent robust world models of how events unfold over time, and this was true from 2019 through early 2023.
No horizon Evidence as of
Limitation: The conclusion depends partly on what counts as robust.
Source: Gaslighting and reality in AI
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Machines will just get more and more reliable and more general.”
Our reading: AI systems will gradually become more reliable and more general.
No deadline stated Evidence as of
Limitation: Benchmark performance is an incomplete measure of generality.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Neither one of us believes for a minute that large language models are remotely close to artificial general intelligence, and what’s more, we mostly agree about what’s missing: semantics, reasoning, common sense, theory of mind and so on.”
Our reading: Large language models are not remotely close to AGI; they lack semantics, reasoning, common sense and theory of mind.
No horizon Evidence as of
Limitation: Historical benchmark results are context, not a complete January 2023 assessment.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“I think that in achieving AGI we will have no choice but to find ways of integrating whatever we can get out of deep learning with explicit knowledge and explicit cognitive models.”
Our reading: Achieving AGI will require integrating deep learning with explicit knowledge and explicit cognitive models.
No deadline stated Evidence as of
Limitation: The available architecture disclosures are insufficient for a definitive architectural test.
Assessment: UnresolvedSaid
Gary Marcus Observation
“A decent fraction of the AI community has already recognized that the primary forte of large language models is in generating bullshit; many also see that the current path to autonomous vehicles is not making the progress that some expected.”
Our reading: A decent fraction of the AI community has already recognized that the primary forte of large language models is generating bullshit, and many also see that the current path to autonomous vehicles is not making the progress that some expected.
No horizon Evidence as of
Limitation: The survey predates the post and covers NLP researchers rather than all AI researchers.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“In a recent Wired article, I predicted that 2023 see the first death attributable to large language models, and I stand by that prediction.”
Our reading: 2023 will see the first death attributable to large language models.
Deadline passed Evidence as of
Limitation: The strongest evidence is original journalism, not a public death-investigation record.
Assessment: UnresolvedSaid
Gary Marcus Observation
“The neurosymbolic AI community is growing fast; conferences that used to be dozens are now thousands.”
Our reading: The neurosymbolic AI community is growing quickly, with conferences going from dozens to thousands of attendees.
No horizon Evidence as of
Limitation: The only attendance figure comes from a single virtual IBM event, reported by IBM itself. It may not be comparable with in-person neurosymbolic workshops.
Assessment: UnresolvedSaid
Gary Marcus Observation
“ChatGPT simply cannot reason well enough to to do this; as I have documented in previous essays, it has demonstrably misunderstood physics, biology, psychology, medicine, and so forth.”
Our reading: ChatGPT cannot reason well enough to make a substantive scientific contribution and has misunderstood physics, biology, psychology, and medicine.
No horizon Evidence as of
Limitation: The evidence does not independently establish misunderstandings in physics and psychology [e1, e2].
Source: Scientists, please don’t let your chatbots grow up to be co-authors
Assessment: UnresolvedSaid
Gary Marcus Observation
“It is not contributing genuine ideas, nor coming up with carefully controlled experimental designs, nor summing up prior literature in thoughtful ways.”
Our reading: ChatGPT does not contribute genuine ideas, design controlled experiments, or thoughtfully summarize literature.
No horizon Evidence as of
Limitation: Proposal ratings do not establish scientific validity, accurate literature synthesis, or sound experimental controls [e1].
Source: Scientists, please don’t let your chatbots grow up to be co-authors
Assessment: UnresolvedSaid
Gary Marcus Observation
“That’s a lot, and not the sort of thing that a rocketship startup would typically agree to.”
Our reading: Giving Microsoft 75% of OpenAI's profits until it recoups its investment is not something a rocketship startup would typically agree to.
No horizon Evidence as of
Limitation: The comparison group and historical window require interpretation.
Assessment: UnresolvedSaid
Gary Marcus Observation
“ChatGPT made all the same kinds of mistakes that its predecessors did.”
Our reading: ChatGPT made all the same kinds of mistakes that earlier GPT-3 versions did.
No horizon Evidence as of
Limitation: Selected failures establish occurrence, not prevalence.
Source: Large Language Models like ChatGPT say The Darnedest Things
Assessment: UnresolvedSaid
Gary Marcus Observation
“ChatGPT couldn’t, for example, reliably count to four (figure 1) or do one-digit arithmetic in the context of simple word problem (figure 2).”
Our reading: ChatGPT could not reliably count to four or do one-digit arithmetic in simple word problems.
No horizon Evidence as of
Limitation: The packet provides no independent repeated or sufficiently varied launch-period tests for either specified task family.
Source: Large Language Models like ChatGPT say The Darnedest Things
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“In our opinion, this was inevitable, in a system that has been trained to optimize the goal of producing one plausible-sounding word after another rather than actually engage with the meaning of language.”
Our reading: ChatGPT's errors were inevitable because it is trained to produce plausible next words rather than engage with meaning.
No horizon Evidence as of
Limitation: The claim combines a causal assertion with an undefined account of meaning.
Source: Large Language Models like ChatGPT say The Darnedest Things
Assessment: UnresolvedSaid
Gary Marcus Observation
“(Spoiler alert: they’re not, at least not yet.)”
Our reading: Current approaches to getting machines to reason about causality are not adequate, at least not yet.
No horizon Evidence as of
Limitation: The cited experiments and theoretical results are historical context, not comprehensive measurements within the target month.
Source: An epic AI Debate—and why everyone should be at least a little bit worried about AI going into 2023
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“But as they do so, as they glom stuff together, something often gets lost in translation: which bits of text do and do not truly belong together.”
Our reading: When LLMs paraphrase and combine text, they often lose track of which pieces truly belong together, producing errors.
No horizon Evidence as of
Limitation: Summarization and truthfulness benchmarks cover different tasks and model populations.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Once again, the result was fluent prose, and once again, hallucination ruled:”
Our reading: YouChat, though tailored for search, produced fluent but hallucinated answers to his friend's queries.
No horizon Evidence as of
Limitation: Unresolved does not mean the reported exchanges were false.
Assessment: UnresolvedSaid
Gary Marcus Observation
“As it happens, large language models are also wicked hard to update, typically requiring full retraining, sometimes taking weeks or months.”
Our reading: LLMs are very hard to update, usually needing full retraining that can take weeks or months.
No horizon Evidence as of
Limitation: Most relevant records describe research methods rather than representative deployment practice.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Which means for example that ChatGPT, released in November, is so old it doesn’t know who owns Twitter:”
Our reading: ChatGPT's knowledge is out of date; it doesn't know who owns Twitter.
No horizon Evidence as of
Limitation: No independent December ownership transcript was verified.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“GPT-4 will be trained on considerably more, a significant fraction of the internet as a whole.”
Our reading: GPT-4 will be trained on considerably more data than GPT-3, a significant fraction of the internet.
No deadline stated Evidence as of
Limitation: This is an evidence gap, not evidence that GPT-4 used little data.
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“GPT-4 may be better at faking term papers, but if it follows the same playbook as it predecessors, it still won’t really understand the world, and seams will eventually show.”
Our reading: If GPT-4 follows its predecessors' approach, it will not truly understand the world and its flaws will eventually show.
No deadline stated Evidence as of
Only if: if it follows the same playbook as its predecessors
Assessment: UnresolvedSaid
Gary Marcus Observation
“As I told NYT columnist Farhad Manjoo, ChatGPT, like earlier, related systems is “still not reliable, still doesn’t understand the physical world, still doesn’t understand the psychological world and still hallucinates.””
Our reading: ChatGPT is not reliable, does not understand the physical or psychological world, and hallucinates.
No horizon Evidence as of
Limitation: The result depends on treating understanding as practical reasoning competence.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Although GPT-4 will definitely seem smarter than its predecessors, its internal architecture remains problematic.”
Our reading: GPT-4 will seem smarter than predecessors but its architecture remains problematic.
No deadline stated Evidence as of
Limitation: The comparative evaluation was conducted by the model's developer.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Which is to say it makes stuff up, regularly; a fair bit of what it is says simply isn’t true.”
Our reading: ChatGPT regularly makes things up and a fair amount of its output is false.
No horizon Evidence as of
Limitation: This does not mean fabrication was absent or rare.
Assessment: UnresolvedSaid
Gary Marcus Forecast
““Scaling” alone—building bigger and models until they absorb the entire internet — will prove useful, but only to a point.”
Our reading: Scaling alone will prove useful only up to a point.
No deadline stated Evidence as of
Limitation: The claim specifies no deadline, capability threshold, or operational test of trustworthy general intelligence.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“they cost almost nothing to operate, and so they are on a path to reducing the cost of generating disinformation to zero.”
Our reading: Because generative AI systems like ChatGPT, Dall-E 2, and Lensa cost almost nothing to operate, they are on a path to reducing the cost of generating disinformation to zero.
No horizon Evidence as of
Limitation: The economically negligible reading of 'zero' is an interpretation.
Source: AI's Jurassic Park moment
Assessment: UnresolvedSaid
Gary Marcus Observation
“they can easily be automated to generate misinformation at unprecedented scale.”
Our reading: Generative AI systems like ChatGPT, Dall-E 2, and Lensa can easily be automated to generate misinformation at unprecedented scale.
No horizon Evidence as of
Limitation: The supplied experiment predates the November–December 2022 target period [e1].
Source: AI's Jurassic Park moment
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“If the website is flooded with worthless code examples, programmers will no longer go there, its database of over 30 million questions and answers will become untrustworthy, and the 14 year old website will die.”
Our reading: If Stack Overflow is flooded with worthless generated code, programmers will leave, its database will become untrustworthy, and the site will die.
No deadline stated Evidence as of
Only if: if the website is flooded with worthless AI-generated code examples
Source: AI's Jurassic Park moment
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“When GPT sounds plausible, it is because every paraphrased bit that it pastes together is grounded in something that actual humans said, and there is often some vague (but often irrelevant) relationship between..”
Our reading: GPT sounds plausible because each piece it combines comes from something humans actually said, and the pieces are often loosely related.
No horizon Evidence as of
Limitation: The meanings of bit, paraphrase and grounded materially affect the test.
Source: How come GPT can seem so brilliant one minute and so breathtakingly dumb the next?
Assessment: UnresolvedSaid
Gary Marcus Observation
“As it turns out, the architecture of Cicero differs profoundly from most of what’s been talked about in recent years in AI.”
Our reading: Cicero's architecture differs profoundly from most AI approaches discussed in recent years.
No horizon Evidence as of
Limitation: Selected examples establish differences, not prevalence.
Source: What does Meta AI’s Diplomacy-winning Cicero Mean for AI?
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“if Alexa could converse freely, user expectations would go through the roof, and mostly be unmeetable.”
Our reading: If Alexa could converse freely, user expectations would rise to a level that mostly could not be met.
No deadline stated Evidence as of
Only if: if Alexa could converse freely
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Turning LLMs into a product that controls your home and talks to you in a way that would be reliable enough to use at scale in millions of homes is still a long, long way away.”
Our reading: A reliable LLM-based conversational home-control product usable at scale in millions of homes is still a long, long way away.
No deadline stated Evidence as of
Limitation: Large-scale participation is company-reported.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Just like every other large language model I have seen.”
Our reading: Like Galactica, every other large language model the author has seen prevaricates (fabricates content) a lot.
No horizon Evidence as of
Limitation: False answers on deliberately challenging questions are not a direct estimate of fabrication in ordinary use.
Source: A Few Words About Bullshit
Assessment: UnresolvedSaid
Gary Marcus Forecast
“High school students will love it, and use it to fool and intimidate (some of) their teachers.”
Our reading: High school students will love Galactica and will use it to fool and intimidate some of their teachers.
No deadline stated Evidence as of
Limitation: A search that finds no qualifying incidents cannot establish that none occurred.
Source: A Few Words About Bullshit
Assessment: UnresolvedSaid
Gary Marcus Forecast
“The problem is about to get much, much worse.”
Our reading: The problem of misinformation on Twitter is about to get much, much worse.
No deadline stated Evidence as of
Limitation: An increase in misinformation exposure could occur without an increase in posting volume.
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“As the pace of machine-generated misinformation picks up, Twitter’s existing effort, Community Notes (aka Birdwatch), which is mostly done manually, by humans, is going to get left in the dust.”
Our reading: As machine-generated misinformation accelerates, Twitter's mostly manual Community Notes will be overwhelmed.
No deadline stated Evidence as of
Only if: as the pace of machine-generated misinformation picks up
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: UnresolvedSaid
Gary Marcus Observation
“Large language models are without equal at generating misinformation, but they suck at detecting it.”
Our reading: LLMs are unmatched at generating misinformation but poor at detecting it.
No horizon Evidence as of
Limitation: No admitted comparison establishes unmatched misinformation generation during the target period.
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Scaling won’t solve this.”
Our reading: Scaling LLMs will not solve their inability to reason about truth and detect misinformation.
No deadline stated Evidence as of
Limitation: Truthful answer generation and misinformation detection are related but distinct tasks.
Source: Bravo @ElonMusk for wanting Twitter to become the world’s “most accurate source of information”
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Scaling maximalism is an interesting hypothesis, but I stand by my prediction that it won’t get us to AGI.”
Our reading: Scaling maximalism will not get us to AGI.
No deadline stated Evidence as of
Limitation: This is not a finding that AGI is absent everywhere; it is a finding that the reviewed evidence does not settle this forecast.
Source: What “game over” for the latest paradigm in AI might look like
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“Any one of them would mean we need a real paradigm shift, if we are to get to AGI.”
Our reading: If any one of the three failure modes (not enough data, not enough compute, tasks that don't scale) holds, reaching AGI will need a real paradigm shift.
No deadline stated Evidence as of
Only if: if any of the three failure modes (data, compute, non-scaling tasks) holds
Source: What “game over” for the latest paradigm in AI might look like
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“His hypothesis, now generally called scaling maximalism, remains extremely popular, in no small part because bigger and bigger models have indeed continued to do ever more impressive things.”
Our reading: Scaling maximalism remains extremely popular, in no small part because bigger and bigger AI models have continued to do ever more impressive things.
No horizon Evidence as of
Limitation: The survey concerns NLP researchers, not all AI researchers, companies, investors, or the public.
Source: What “game over” for the latest paradigm in AI might look like
Assessment: UnresolvedSaid
Gary Marcus Observation
“As I have long argued, capturing this without cognitive models and common sense is really hard.”
Our reading: Capturing the pragmatic implications of language without cognitive models and common sense is really hard.
No horizon Evidence as of
Limitation: The finding does not deny the demonstrated difficulty.
Source: What “game over” for the latest paradigm in AI might look like
Assessment: UnresolvedSaid
Gary Marcus Observation
“It’s also, as ever, stuck in the liminal space between superintelligence and super-lost.”
Our reading: Make a Video, as with AI systems before it, is caught between very impressive output and basic errors, such as giving a cat a human hand or a painting that never changes.
No horizon Evidence as of
Limitation: The original moving clips were not independently inspected in this assessment.
Assessment: UnresolvedSaid
Gary Marcus Forecast
“This is the uncanny valley we are in, and one that we occupy for a while.”
Our reading: AI systems such as video, image, and language generators are now in an uncanny valley of impressive output with basic errors, and will stay there for a while.
No deadline stated Evidence as of
Limitation: This outcome does not mean the observed persistence is unsupported; it reflects the missing duration and resolution threshold.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“We in AI have a long long way to go. Data and compute alone won’t be enough.”
Our reading: AI has a long way to go, and data and compute alone will not be enough.
No horizon Evidence as of
Limitation: Benchmark performance covers only part of the capability claim.
Source: Sub-Optimal
Assessment: UnresolvedSaid
Gary Marcus Observation
“Musk barely even acknowledged the problem, and certainly didn’t lay out any sort of credible strategy for attacking it..”
Our reading: Musk barely acknowledged the common-sense problem and laid out no credible strategy for addressing it.
No horizon Evidence as of
Limitation: An absence claim requires stronger coverage than targeted keyword searches.
Source: Sub-Optimal
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“The reality is that AI needs genuine, paradigm-shift level innovation. Simply building ever bigger neural networks won’t cut it.”
Our reading: AI needs paradigm-shift innovation; simply scaling neural networks will not be enough.
No horizon Evidence as of
Limitation: The capability target and paradigm boundary are not operationally defined.
Source: Sub-Optimal
Assessment: UnresolvedSaid
Gary Marcus Observation
“Second, there was very little vision for how Tesla would build the cognitive part of the AI they will need, beyond the basics of motor control (which Boston Dynamics already does so well), nor much recognition about why robotics is so hard in the real world.”
Our reading: Tesla showed very little vision for building the cognitive AI Optimus needs beyond motor control, and little recognition of why real-world robotics is hard.
No horizon Evidence as of
Limitation: The cited notes are secondary and include automated transcription.
Source: Sub-Optimal
Assessment: UnresolvedSaid
Gary Marcus Observation
“Musk is almost never right about deadlines.”
Our reading: Elon Musk's deadlines for technology projects are almost never met.
No horizon Evidence as of
Limitation: The usable evidence is concentrated on Tesla.
Source: Top 5 reasons Tesla’s Optimus announcement may turn out be a dud
Assessment: UnresolvedSaid
Gary Marcus Observation
“On #1, the answer is a clear yes; only highly trained human artists could do better.”
Our reading: Image synthesis systems generate high quality images that only highly trained human artists could surpass.
No horizon Evidence as of
Limitation: Image quality can vary by task and evaluator.
Source: Form, function, and the giant gulf between drawing a picture and understanding the world
Assessment: UnresolvedSaid
Gary Marcus Observation
“Nine more tries, and only one very marginal success (top right corner):”
Our reading: Stable Diffusion produced only one very marginal success in nine further attempts at a part-function prompt.
No horizon Evidence as of
Limitation: Success grading includes a subjective boundary.
Source: Form, function, and the giant gulf between drawing a picture and understanding the world
Assessment: UnresolvedSaid
Gary Marcus Observation
“Even “draw a white bicycle with green wheels”, which focuses purely on part-whole relationships without function or complex syntax, is problematic:”
Our reading: Stable Diffusion has trouble even with simple part-whole prompts such as a white bicycle with green wheels.
No horizon Evidence as of
Limitation: Failure to locate independent records does not show that Marcus's displayed results were inaccurate.
Source: Form, function, and the giant gulf between drawing a picture and understanding the world
Assessment: UnresolvedSaid
Gary Marcus Observation
“Negation is, as ever, a problem.”
Our reading: Text-to-image systems like Stable Diffusion continue to have problems with negation in prompts.
No horizon Evidence as of
Limitation: This is an evidence gap, not a finding that negation worked reliably.
Source: Form, function, and the giant gulf between drawing a picture and understanding the world
Assessment: UnresolvedSaid
Gary Marcus Observation
“Google cloaks its current work the trappings of science (article preprints with citations and graphs and so forth) but when it comes to community inspection, they’ve systematically opted out.”
Our reading: Google has systematically refused to let the scientific community inspect models such as Imagen and LaMDA.
No horizon Evidence as of
Limitation: The evidence distinguishes public availability from scientific access but does not fully document the latter.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: UnresolvedSaid
Gary Marcus Observation
“Notwithstanding the rumors about Google Imagen, nobody has yet publicly demonstrated a machine that can relate the meanings of sentences to their parts the way a five-year-old child can.”
Our reading: No one, Imagen included, has publicly shown a machine that relates sentence meanings to their parts as well as a five-year-old.
No horizon Evidence as of
Limitation: The named age threshold is not measured by the cited benchmark.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: UnresolvedSaid
Gary Marcus Observation
“But many people have claimed over the years to solve the problem, and none of those systems have proven to be reliable; every proposed solution has been like Clever Hans, working in dim light, leveraging large databases to some degree, but falling apart upon careful inspection; they might get 60% on some task, but they never really master it.”
Our reading: Every claimed solution to compositionality so far has proven unreliable and falls apart on careful inspection.
No horizon Evidence as of
Limitation: The universal scope exceeds the coverage of the retrieved evaluations.
Source: Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
Assessment: UnresolvedSaid
Gary Marcus Forecast
“Cool demo, yes. Real world product, anytime soon? Fuggedaboutit.”
Our reading: PaLM-SayCan will not become a real-world product anytime soon.
No deadline stated Evidence as of
Limitation: The search cannot prove the absence of every derivative.
Assessment: UnresolvedSaid
Gary Marcus Observation
“Facebook M was canceled, and 7 years later nobody yet can buy a super-smart (or even decently smart) robot at any price.”
Our reading: As of August 2022, nobody can buy even a decently smart robot at any price.
No horizon Evidence as of
Limitation: The sources do not provide a comprehensive market record for August 18, 2022 [e1][e2][e3].
Assessment: UnresolvedSaid
Gary Marcus Observation
“The thing is, current AI systems that (roughly speaking) do a lot of cutting and pasting without those grammar rules and trees aren’t really all that close to understanding human language.”
Our reading: Current AI systems that, roughly speaking, do a lot of cutting and pasting without grammar rules and trees aren't really all that close to understanding human language.
No horizon Evidence as of
Limitation: Benchmark achievement and general language comprehension are different claims.
Assessment: UnresolvedSaid
Gary Marcus Observation
“We don’t know how kids do it; we do know that what we are doing with machines currently isn’t really working.”
Our reading: Current machine approaches to language acquisition are not really working.
No horizon Evidence as of
Limitation: The selected tasks cover components of language competence.
Assessment: UnresolvedSaid
Gary Marcus Observation
“All normal children manage to acquire language in this sense; no existing machine does.”
Our reading: No existing machine has acquired language in the sense of mapping sentences to meanings and meanings to sentences.
No horizon Evidence as of
Limitation: Formal meaning representations cover only selected aspects of meaning.
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“In our view, inadequate attention to these three factors has serious consequences, including:”
Our reading: Neglecting reference, cognitive models and compositionality causes serious problems in large language models.
No horizon Evidence as of
Limitation: The search is not an exhaustive systematic review.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Observation
“here, DALL-E mini struggles to accurately represent the implicit action for the woman, capturing the apple but not reliably capturing the eating, often sticks an apple in the man’s mouth rather than the sandwich, and in most cases misses the implication that the man and woman are supposed to be eating different things.”
Our reading: In the authors' experiments with the sentence 'The man is eating a sandwich and the woman an apple', DALL-E Mini struggles to represent the woman's implied action: it captures the apple but not reliably the eating, often puts an apple in the man's mouth instead of the sandwich, and in most cases misses that the two are eating different things.
No horizon Evidence as of
Limitation: A replication could establish reproducibility without proving what happened in the authors' experiment.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“The fact that its word sequence predictions are often so good tempts many into ascribing meaning, but this is more a reflection of the tremendous size of the input corpus than of a system that genuinely maps sentence onto reference and cognitive models.”
Our reading: GPT-3's good word predictions come more from its huge training corpus than from any real mapping of sentences onto reference and cognitive models.
No horizon Evidence as of
Limitation: Results from smaller transformers, BART, and T5 are context only.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Observation
“Similarly, consider pronouns like “himself”: DALL-E does not do well with sentences like “The man is painting a picture of himself”.”
Our reading: DALL-E does poorly on sentences with reflexive pronouns such as 'The man is painting a picture of himself'.
No horizon Evidence as of
Limitation: Model identity and the standard for poor performance remain unspecified.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Observation
“Compositionality has long been a central concept in linguistics and philosophy (even Aristotle acknowledged it), yet so-called foundation models (large language models, GPT-3, BERT, deep learning approaches) sidestep it.”
Our reading: Foundation models such as GPT-3 and BERT sidestep compositionality.
No horizon Evidence as of
Limitation: The studies use different tasks and assistance levels, preventing a direct comparison [e1, e2].
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“(a) the tendency of large language models to lose coherence over time, drifting into “empty” language with no clear connection to reality;”
Our reading: Neglecting reference, cognitive models and compositionality leads LLMs to lose coherence over time and drift into empty language with no clear connection to reality.
No horizon Evidence as of
Limitation: TattleTale has limited empirical evaluation.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“(b) the difficulty of large language models in distinguishing truth from falsehoods;”
Our reading: Neglecting reference, cognitive models and compositionality makes it hard for LLMs to tell truth from falsehood.
No horizon Evidence as of
Limitation: TruthfulQA's questions target misconceptions; its results do not establish an overall factual-error rate [e8].
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Observation
“But the interesting thing to note here is that we humans can choose to interpret language compositionally, or not—this degree of cognitive flexibility is really quite impressive, but lacking in current large language models.”
Our reading: Current LLMs lack the human ability to choose whether to interpret language compositionally.
No horizon Evidence as of
Limitation: The available tests use different tasks and model configurations.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Observation
“Language is about a connection between words (or sentence) and the world; the sequences of words that large language models utter lack connection to the external world.”
Our reading: The word sequences that large language models produce have no connection to the external world, so they lack reference.
No horizon Evidence as of
Limitation: Source attribution does not by itself establish situated reference or factual correctness [e3].
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Observation
“In such cases, DALL-E Mini typically generates images of bowls of cucumbers, and only cucumbers, leaving out the tomatoes altogether (where a human might reasonably expect there to be at least two, given the plural).”
Our reading: Given comparative prompts such as 'more cucumbers than tomatoes', DALL-E Mini usually leaves out the tomatoes entirely.
No horizon Evidence as of
Limitation: The evidence establishes neither a majority omission rate nor a rate sufficient to contradict the claim.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“Unless the field can find a way to tightly integrate large language models with cognitive models and reference, we expect that the field will eventually hit a dead end, particularly in the understanding long discourses, and series of events unfolding over time, and that the problem of misinformation will remain unsolved.”
Our reading: Unless LLMs are tightly integrated with cognitive models and reference, the field will eventually hit a dead end, especially in understanding long discourse and events over time, and misinformation will remain unsolved.
No deadline stated Evidence as of
Only if: unless LLMs are tightly integrated with cognitive models and reference
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“(c) the struggle in these models to avoid perpetuating bias and toxic speech.”
Our reading: Neglecting reference, cognitive models and compositionality makes it hard for LLMs to avoid repeating bias and toxic speech.
No horizon Evidence as of
Limitation: The search is not proof that no relevant experiment exists.
Source: Three ideas from linguistics that everyone in AI should know
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“As far as I can tell, nobody has a real solution to the outlier problem of coping with unusual circumstances; without it, full self driving will remain a pipe dream.”
Our reading: Without a solution to the outlier problem, full self-driving will not be achieved.
No deadline stated Evidence as of
Only if: without a solution to the outlier problem
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“👉an explicit type/token distinction will be crucial”
Our reading: Whenever AGI arrives, an explicit type/token distinction will be crucial to it.
No deadline stated Evidence as of
Only if: whenever AGI comes
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“👉explicit cognitive models will be crucial”
Our reading: Whenever AGI arrives, explicit cognitive models will be crucial to it.
No deadline stated Evidence as of
Only if: whenever AGI comes
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“👉operations over variables (including storing, retrieving and comparing values) will be crucial”
Our reading: Whenever AGI arrives, operations over variables, including storing, retrieving and comparing values, will be crucial to it.
No deadline stated Evidence as of
Only if: whenever AGI comes
Assessment: UnresolvedSaid
Gary Marcus Observation
“Large language models are (unless hybridized with symbolic systems, a la Jurassic X) are, on their own, inherently weak at understanding the physical and psychological world.”
Our reading: LLMs on their own, without symbolic hybridization, are inherently weak at understanding the physical and psychological world.
No horizon Evidence as of
Limitation: TRIP tests a narrower model population than the frontier generative systems discussed in the post.
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Gary Marcus Observation
“Interestingly, as Ernie has just discovered, across the full set of our Technology Review questions the difference from GPT-3’s earlier version to the current version (InstructGPT) is far more modest than one might have expected: fewer outright wacky answers, some old answers now correct, but some backsliding, too.”
Our reading: As Davis has found, across the full set of their Technology Review questions, the difference between GPT-3's earlier version and InstructGPT is far more modest than one might have expected: fewer outright wacky answers and some old answers now correct, but some backsliding too.
No horizon Evidence as of
Limitation: The source restriction excludes the report most directly associated with the claim.
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Gary Marcus Observation
“The core claims that Ernie and I made in Technology Review still all, so far as I can tell, hold true, nearly two years and many even larger models later. Yes, some of the specific examples we pointed out have been remedied, but the general problems have not.”
Our reading: So far as Marcus can tell, all the core claims he and Davis made in Technology Review still hold true nearly two years and many even larger models later; some of their specific examples have been remedied, but the general problems have not.
No horizon Evidence as of
Limitation: This is a gap in the retrieved evidence, not evidence that the categories had been solved.
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“But they still don’t convey genuine comprehension, and so they are still very easy for Ernie and me (or anyone else who cares to try) to break.”
Our reading: Newer LLMs still lack genuine comprehension and are therefore still very easy for anyone to break.
No horizon Evidence as of
Limitation: Benchmark scores and qualitative examples do not establish that anyone could easily elicit failures across all relevant models [e1, e2].
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“I am willing to bet SlateStarCodex now (terms to be negotiated) that if OpenAI gives us unrestricted access to GPT-4, whenever that is released, and assuming that is basically the same architecture but with more data, that within a day of playing around with it, Ernie and I will still be able find lots of examples of failures in physical reasoning, temporal reasoning, causal reasoning, and so forth.”
Our reading: If given unrestricted access to GPT-4 and it is basically the same architecture with more data, Marcus and Davis will find many failures in physical, temporal, and causal reasoning within a day.
No deadline stated Evidence as of
Only if: OpenAI gives unrestricted access to GPT-4
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“The improvements, such as they are, come, primarily because the newer models have larger and larger sets of data about how human beings use word sequences, and bigger word sequences are certainly helpful for pattern matching machines.”
Our reading: The newer models' improvements, such as they are, come primarily from their larger and larger sets of data about how humans use word sequences, which are certainly helpful for pattern-matching machines.
No horizon Evidence as of
Limitation: An optimal scaling prescription is not a decomposition of historical progress.
Source: What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus
Assessment: UnresolvedSaid
Gary Marcus Observation
“The fourth thing to realize is that we, and by this I mean humanity as a whole, still don’t really have an adequate methodology for building complex cognitive systems.”
Our reading: Humanity lacks an adequate methodology for building complex cognitive systems.
No horizon Evidence as of
Limitation: The empirical study covers a sample of practitioners and systems.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Gary Marcus Observation
“I don’t know what you have planned for Optimus, but I can promise you that the AGI that you would need for a general-purpose domestic robot (where every home is different, and each poses its own safety risks) is way beyond what you would need for a car that drives on roads that are more or less engineered the same way from one town to the next.”
Our reading: The AGI needed for a general-purpose domestic robot, where every home is different and poses its own safety risks, is way beyond what is needed for a car driving on roads engineered more or less the same from town to town.
No horizon Evidence as of
Limitation: Household robotics evidence includes simulation and constrained kitchen tasks.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Gary Marcus Observation
“We still don’t have anything like stable or trustworthy solutions for common sense, reasoning, language, or analogy. Truth is, I have been using this same pie chart for about 5 years, and the situation has scarcely changed.”
Our reading: AI has no stable or trustworthy solutions for common sense, reasoning, language or analogy, and this has scarcely changed in about five years.
No horizon Evidence as of
Limitation: The studies use different tasks, models and evaluation methods.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Gary Marcus Observation
“Deep learning is pretty decent at, for example, recognizing objects, but not nearly as good at planning, or reading, or language comprehension.”
Our reading: Deep learning is decent at object recognition but much worse at planning, reading and language comprehension.
No horizon Evidence as of
Limitation: Benchmark success does not establish broad real-world comprehension.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“The thing is, it’s easy to convince yourself that AI problems are much easier than they are actually are, because of the long tail problem.”
Our reading: The long tail problem makes it easy to believe AI problems are much easier than they actually are.
No horizon Evidence as of
Limitation: The closest empirical evidence uses a simulated robot task.
Source: Dear Elon Musk, here are five things you might want to consider about AGI
Assessment: UnresolvedSaid
Gary Marcus Observation
“As Ernie Davis, Scott Aaronson, and I showed, a week or so after the release, it’s easily fooled, especially with complex sentences and words like “similar” that would require a deep understanding of language to be interpreted correctly”
Our reading: DALL-E 2 is easily fooled, especially by complex sentences and words requiring deep language understanding.
No horizon Evidence as of
Limitation: Unresolved does not mean the claim was false.
Source: Horse rides astronaut
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“Since we know that Imagen can draw images of horses riding astronauts (if the instructions are specific enough) we know that the failure to draw a horse riding an astronaut given the prompt “a horse riding an astronaut” can only have one explanation: the system doesn’t understand what the prompt “a horse riding an astronaut” means.”
Our reading: Since Imagen can draw horses riding astronauts when the instructions are specific enough, its failure to draw one from the prompt 'a horse riding an astronaut' can only be explained by the system not understanding what that prompt means.
No horizon Evidence as of
Limitation: The available demonstration is not a controlled causal experiment.
Source: Horse rides astronaut
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“And, no, it wasn’t about common sense at all.”
Our reading: Imagen's failure on 'a horse riding an astronaut' was not due to common sense.
No horizon Evidence as of
Limitation: The demonstration provides no repeated-sample success rates or controlled comparison separating wording from scene plausibility [e5].
Source: Horse rides astronaut
Assessment: UnresolvedSaid
Gary Marcus Observation
“(In reality, Imagen has no idea what weight, mass, or density are.)”
Our reading: Imagen has no understanding of weight, mass or density.
No horizon Evidence as of
Limitation: The behavioral interpretation makes the claim testable but does not settle philosophical questions about understanding.
Source: Horse rides astronaut
Assessment: UnresolvedSaid
Gary Marcus Causal claim
“There is no intention to create misinformation—but also no capacity to avoid it, because fundamentally, GPT-3 is a model of how words relate to one another, not a model of how language might relate to the perceived world.”
Our reading: GPT-3 cannot avoid producing misinformation because it models how words relate to each other, not how language relates to the world.
No horizon Evidence as of
Limitation: The evidence is about error rates, not about mechanism.
Source: Noam Chomsky and GPT-3
Assessment: UnresolvedSaid
Gary Marcus Conditional forecast
“The bottom line is this, something that AI once cherished but has now forgotten: If we are to build AGI, we are going to need to learn something from humans, how they reason and understand the physical world, and how they represent and acquire language and complex concepts.”
Our reading: If we are to build AGI, we will need to learn something from humans: how they reason and understand the physical world, and how they represent and acquire language and complex concepts.
No deadline stated Evidence as of
Only if: if we are to build AGI
Assessment: UnresolvedSaid
Gary Marcus Observation
“As I have written, I am fine with thinking of Gato as an “Alt Intelligence”— an interesting exploration in alternative ways to build intelligence—but we need to take it in context: it doesn’t work like the brain, it doesn’t learn like a child, it doesn’t understand language, it doesn’t align with human values, and it can’t be trusted with mission-critical tasks.”
Our reading: Gato does not work like the brain, learn like a child, understand language, or align with human values, and cannot be trusted with mission-critical tasks.
No horizon Evidence as of
Limitation: The evidence is the developers' original report, rather than independent validation of all five assertions [e6].