Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Claims checked against evidence by Gary Marcus, page 19 of 19

Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.

Claims checked against evidence by Gary Marcus (1870)

Each of these claims was checked against dated evidence.

Page 19 of 19: claims 1801 to 1870 of 1870, ordered by the date first said, newest first. To find a claim by its words, use search.

  1. Assessment: UnresolvedSaid

    “Similarly, consider pronouns like “himself”: DALL-E does not do well with sentences like “The man is painting a picture of himself”.”

  2. Assessment: UnresolvedSaid

    “Compositionality has long been a central concept in linguistics and philosophy (even Aristotle acknowledged it), yet so-called foundation models (large language models, GPT-3, BERT, deep learning approaches) sidestep it.”

  3. Assessment: SupportedSaid

    “Since then, the results on benchmarks have gotten better, but there’s still something missing.”

    What happened: PaLM's contemporary report documents continued benchmark improvements. This supports the improvement component, without establishing that benchmark success implies general understanding. (arXiv; Chowdhery et al., )

  4. Assessment: ContradictedSaid

    “(LLM’s like GPT produce well-formed prose but do not produce interpretable representations of utterances that reflect structured relationships between the parts of those sentences.)”

    What happened: The study uses late-2021 GPT-3 and Codex models to map utterances into structured meaning representations. It demonstrates a task capability, while its use of prompts and decoding machinery limits conclusions about ordinary internal representations. (arXiv; Richard Shin and Benjamin Van Durme, )

  5. Assessment: UnresolvedSaid

    “(a) the tendency of large language models to lose coherence over time, drifting into “empty” language with no clear connection to reality;”

  6. Assessment: SupportedSaid

    “noncanonical reorderings such as these remain problematic for current AI approaches.)”

    What happened: Winoground requires matching two images to captions containing the same words in different orders. Its poor group scores across contemporary vision-language models support substantial difficulty preserving the changed relationships. (Thrush et al., CVPR 2022 / arXiv, )

  7. Assessment: UnresolvedSaid

    “(b) the difficulty of large language models in distinguishing truth from falsehoods;”

  8. Assessment: UnresolvedSaid

    “But the interesting thing to note here is that we humans can choose to interpret language compositionally, or not—this degree of cognitive flexibility is really quite impressive, but lacking in current large language models.”

  9. Assessment: UnresolvedSaid

    “Language is about a connection between words (or sentence) and the world; the sequences of words that large language models utter lack connection to the external world.”

  10. Assessment: ContradictedSaid

    “Current systems simply do not postulate such cognitive models—and so, they technically can’t prove themselves wrong, because they do not even have a representation of what being “right” would look like.”

    What happened: The experiments evaluate models critiquing their own outputs against supplied passages or task constraints, and using critiques to improve answers. This contradicts the functional claim of complete inability, while leaving the nature of internal cognitive representations unsettled. (arXiv; William Saunders and colleagues, OpenAI, )

  11. Assessment: ContradictedSaid

    “The downstream consequence of all of this is that these models have no grasp whatsoever on truth.”

    What happened: GPT-judge classified the factual truth of answers and transferred across held-out answer-producing model families. This contradicts absolute behavioral incapacity, but all benchmark questions were represented in training, so it does not establish unrestricted factual understanding. (Association for Computational Linguistics; Stephanie Lin, Jacob Hilton and Owain Evans; ar5iv mirror, )

  12. Assessment: UnresolvedSaid

    “In such cases, DALL-E Mini typically generates images of bowls of cucumbers, and only cucumbers, leaving out the tomatoes altogether (where a human might reasonably expect there to be at least two, given the plural).”

  13. Assessment: UnresolvedSaid

    “Unless the field can find a way to tightly integrate large language models with cognitive models and reference, we expect that the field will eventually hit a dead end, particularly in the understanding long discourses, and series of events unfolding over time, and that the problem of misinformation will remain unsolved.”

  14. Assessment: UnresolvedSaid

    “(c) the struggle in these models to avoid perpetuating bias and toxic speech.”

  15. Assessment: SupportedSaid

    “Because it shows that the people who really claim to be working on full self-driving (which Tesla waffles on1) are having plenty of problems of their own.”

    What happened: The record confirms repeated ADS crash involvement in the initial reporting period. Its abstract defines reportability by ADS use within 30 seconds of a crash, not by a finding of ADS fault. (Transportation Research Board, summarizing NHTSA, )

  16. Assessment: UnresolvedSaid

    “As far as I can tell, nobody has a real solution to the outlier problem of coping with unusual circumstances; without it, full self driving will remain a pipe dream.”

  17. Assessment: SupportedSaid

    “Level 5 has to handle it all; everyday stopped vehicles, weird outliers and so forth. We’re just not there.”

    What happened: The contemporaneous report directly addresses the claimed industry-wide state. Reputable reporting is used because no readable primary source establishing that complete industry-wide negative was obtained. (The Verge, )

  18. Assessment: SupportedSaid

    “Indeed, the NHTSA report data show that even some pretty ordinary circumstances, like avoiding stopped vehicles, are continuing to prove problematic.”

    What happened: The surrounding summary identifies these as additional Autopilot collisions with stopped first-responder vehicles. Its incident list dates Orlando to August 2021, Petaluma to September 2021, and Desert Center to January 2022; the event date records the latest of these in-window incidents. (National Highway Traffic Safety Administration, )

  19. Assessment: UnresolvedSaid

    “We in the AI community have our differences, but pretty much all of find the notion that LaMDA might be sentient completely ridiculous.”

  20. Assessment: UnresolvedSaid

    “The public knows the test of course, but the AI community wishes it would go away; we all know that beating that test isn’t meaningful.”

  21. Assessment: ContradictedSaid

    “Software like LaMDA simply doesn’t; it doesn’t even try to connect to the world at large, it just tries to be the best version of autocomplete it can be, by predicting what words best fit a given context.”

    What happened: The paper describes retrieval, calculator and translation tools, and training that routes queries to those tools during inference. This directly contradicts the assertion that LaMDA does not even attempt external grounding. (Google Research authors / arXiv, )

  22. Assessment: ContradictedSaid

    “But 8 years later I doubt most people (even in AI) have ever even heard of his program, outside of my mentioning it here. It made zero lasting contribution to AI.”

    What happened: The author's statement says Emelya was based on a reworked natural-language analysis system created for Eugene Goostman. Her account dates development to 2016, release to May 2017 and further user research to 2018, documenting a technical legacy beyond the 2014 competition. (Habr / Ontico, account by project participant Anna Degteva, )

  23. Assessment: SupportedSaid

    “The basic finding was: loads of things scale, but not at all ( as foreseen both by Kurenkov in his essay last year and in my much lampooned but basically accurate essay, Deep learning is hitting a wall).”

    What happened: The original report directly states the coexistence Marcus describes. Sections 3.4.2 and 6.1 also explain that metric choice can conceal gradual progress and that some limitations may change with larger models or improved data. (Srivastava et al.; arXiv, )

  24. Assessment: UnresolvedSaid

    “👉an explicit type/token distinction will be crucial”

  25. Assessment: UnresolvedSaid

    “👉explicit cognitive models will be crucial”

  26. Assessment: SupportedSaid

    “The massive paper looks at scaling on many measures, and sees progress on some– but not others. The stuff that isn’t scaling is the “wall”.”

    What happened: The report directly confirms improving and non-improving task scores across evaluated scales. Its discussion of smoother metrics limits what those flat scores imply about underlying capabilities. (Srivastava et al.; arXiv, )

  27. Assessment: UnresolvedSaid

    “👉operations over variables (including storing, retrieving and comparing values) will be crucial”

  28. Assessment: SupportedSaid

    “so many of the unfortunate consequences of a webscraping/big data approach to AI are disproportionately borne by women and minorities.”

    What happened: The audit reports removal rates of 42% for African American English and 32% for Hispanic-aligned English, against 6.2% for White-aligned English. This documents unequal exclusion from a web-scraped training corpus, using dialect as a proxy rather than identifying authors. (Dodge et al.; EMNLP 2021, arXiv version, )

  29. Assessment: SupportedSaid

    “I would be very surprised if that fundamental lack of reliability did not also hold true for GPT-4. Maybe it will take three paragraphs in a prompt rather than one to break it, but a big Transformer alone does not a a theory of the world create.”

    What happened: OpenAI's launch-era report explicitly acknowledges both predicted failure classes and advises safeguards where reliability matters. It also reports substantial factuality improvement, which is compatible with Marcus's allowance for fewer errors. (OpenAI / arXiv, )

  30. Assessment: UnresolvedSaid

    “Large language models are (unless hybridized with symbolic systems, a la Jurassic X) are, on their own, inherently weak at understanding the physical and psychological world.”

  31. Assessment: UnresolvedSaid

    “Interestingly, as Ernie has just discovered, across the full set of our Technology Review questions the difference from GPT-3’s earlier version to the current version (InstructGPT) is far more modest than one might have expected: fewer outright wacky answers, some old answers now correct, but some backsliding, too.”

  32. Assessment: UnresolvedSaid

    “The core claims that Ernie and I made in Technology Review still all, so far as I can tell, hold true, nearly two years and many even larger models later. Yes, some of the specific examples we pointed out have been remedied, but the general problems have not.”

  33. Assessment: UnresolvedSaid

    “But they still don’t convey genuine comprehension, and so they are still very easy for Ernie and me (or anyone else who cares to try) to break.”

  34. Assessment: UnresolvedSaid

    “I am willing to bet SlateStarCodex now (terms to be negotiated) that if OpenAI gives us unrestricted access to GPT-4, whenever that is released, and assuming that is basically the same architecture but with more data, that within a day of playing around with it, Ernie and I will still be able find lots of examples of failures in physical reasoning, temporal reasoning, causal reasoning, and so forth.”

  35. Assessment: UnresolvedSaid

    “The improvements, such as they are, come, primarily because the newer models have larger and larger sets of data about how human beings use word sequences, and bigger word sequences are certainly helpful for pattern matching machines.”

  36. Assessment: UnresolvedSaid

    “The fourth thing to realize is that we, and by this I mean humanity as a whole, still don’t really have an adequate methodology for building complex cognitive systems.”

  37. Assessment: UnresolvedSaid

    “In 2029, AI will not be able to reliably construct bug-free code of more than 10,000 lines from natural language specification or by interactions with a non-expert user.”

  38. Assessment: SupportedSaid

    “You said, for instance in 2015, that (truly) self-driving cars were two years away; you’ve pretty much said the same thing every year since. It still hasn’t happened.”

    What happened: The original interview establishes the attributed prediction. Musk separately allowed additional time for regulatory approval, so the prediction concerned technical capability first. (Fortune, )

  39. Assessment: UnresolvedSaid

    “I don’t know what you have planned for Optimus, but I can promise you that the AGI that you would need for a general-purpose domestic robot (where every home is different, and each poses its own safety risks) is way beyond what you would need for a car that drives on roads that are more or less engineered the same way from one town to the next.”

  40. Assessment: UnresolvedSaid

    “In 2029, AI will not be able to read a novel and reliably answer questions about plot, character, conflicts, motivations, etc.”

  41. Assessment: UnresolvedSaid

    “We still don’t have anything like stable or trustworthy solutions for common sense, reasoning, language, or analogy. Truth is, I have been using this same pie chart for about 5 years, and the situation has scarcely changed.”

  42. Assessment: UnresolvedSaid

    “Deep learning is pretty decent at, for example, recognizing objects, but not nearly as good at planning, or reading, or language comprehension.”

  43. Assessment: UnresolvedSaid

    “In 2029, AI will not be able to take arbitrary proofs from the mathematical literature written in natural language and convert them into a symbolic form suitable for symbolic verification.”

  44. Assessment: SupportedSaid

    “Even within perception 3D perception remains a challenge, and scene understanding is by no means solved.”

    What happened: Testing fifteen common corruptions exposed substantial robustness weaknesses in leading 3D recognition models. The authors also report mitigation methods, so the result establishes remaining challenges rather than an absence of progress. (arXiv; Jiachen Sun and colleagues, )

  45. Assessment: SupportedSaid

    “But for the kinds of AI techniques that are currently popular, and which rely more on big data than on reasoning, it’s a very serious issue.”

    What happened: The contemporary synthesis documents poor learning of rare classes and substantial limitations in conventional deep-learning methods. It also reviews mitigation methods, supporting a serious practical problem without implying that every method fails. (International Journal of Computer Vision; author manuscript rendered by ar5iv, )

  46. Assessment: UnresolvedSaid

    “In 2029, AI will not be able to work as a competent cook in an arbitrary kitchen (extending Steve Wozniak’s cup of coffee benchmark).”

  47. Assessment: MixedSaid

    “Debugging with deep learning is wicked hard, because nobody (a) really understands how it works, and (b) knows how to fix problems other to collect more data and add more layers and so forth.”

    What held: The survey supports the narrower causal point that opacity makes locating and repairing problems harder. Its review of explanation-based debugging also shows that researchers were already developing remedies. (arXiv; Piyawat Lertvittayakumjorn and Francesca Toni, )

    What did not: The authors used causal interventions to investigate factual recall and evaluated targeted factual edits on CounterFact. This supplies a concrete pre-post example of useful mechanistic understanding and repair without adding data or layers. (arXiv; Kevin Meng, David Bau, Alex Andonian and Yonatan Belinkov, )

  48. Assessment: UnresolvedSaid

    “The thing is, it’s easy to convince yourself that AI problems are much easier than they are actually are, because of the long tail problem.”

  49. Assessment: SupportedSaid

    “And it’s why I absolutely guarantee you won’t be shipping level 5 self-driving cars this year or next, no matter what you might have told Chris Anderson at TED.”

    What happened: The annual filing describes Tesla's offerings as driver assistance and says the driver remains responsible for controlling the vehicle. That capability does not meet Level 5. (Tesla, Inc., mirrored by AnnualReports.com, )

  50. Assessment: UnresolvedSaid

    “In 2029, AI will not be able to watch a movie and tell you accurately what is going on (what I called the comprehension challenge in The New Yorker, in 2014).”

  51. Assessment: SupportedSaid

    “But you and I both know that they continue to be a major problem that as yet has no known robust solution.”

    What happened: The survey identifies unresolved obstacles involving graph structure, differing tasks, and different kinds of distribution shift. It also describes progress, supporting the distinction between useful remedies and a general solution. (arXiv; Haoyang Li and colleagues, )

  52. Assessment: SupportedSaid

    “That’s where the deep understanding of language should live—but, despite the PR, it isn’t there.”

    What happened: The original authors identify a category-level alignment weakness in the figure containing the horse-riding-an-astronaut prompt. This supports the contextual reading that reliable compositional instruction understanding was missing. (Google Research, Brain Team; copy hosted by 3DVAR, )

  53. Assessment: UnresolvedSaid

    “As Ernie Davis, Scott Aaronson, and I showed, a week or so after the release, it’s easily fooled, especially with complex sentences and words like “similar” that would require a deep understanding of language to be interpreted correctly”

  54. Assessment: UnresolvedSaid

    “Since we know that Imagen can draw images of horses riding astronauts (if the instructions are specific enough) we know that the failure to draw a horse riding an astronaut given the prompt “a horse riding an astronaut” can only have one explanation: the system doesn’t understand what the prompt “a horse riding an astronaut” means.”

  55. Assessment: UnresolvedSaid

    “And, no, it wasn’t about common sense at all.”

  56. Assessment: UnresolvedSaid

    “(In reality, Imagen has no idea what weight, mass, or density are.)”

  57. Assessment: UnresolvedSaid

    “Furthermore, there is no step in the process to try to derive such structures; so it is just wishful thinking to expect them to be fully sensitive to semantics as it relates to syntactic structure.”

  58. Assessment: ContradictedSaid

    “They do not gravitate towards human language, and they never develop a cohesive undestanding of the world, no matter how much experience they have.”

    What happened: Kallini and colleagues compared GPT-2 learning across English and systematically altered languages, including different training stages. The observed preference conflicts with the language-learning component under the stated family-level interpretation. (arXiv, )

  59. Assessment: UnresolvedSaid

    “There is no intention to create misinformation—but also no capacity to avoid it, because fundamentally, GPT-3 is a model of how words relate to one another, not a model of how language might relate to the perceived world.”

  60. Assessment: UnresolvedSaid

    “Virtually none of the critical problems for AI that cognitive scientists such as Steve Pinker, Judea Pearl, the late Jerry Fodor, and myself have been pointing to for decades is actually solved yet.”

  61. Assessment: UnresolvedSaid

    “The bottom line is this, something that AI once cherished but has now forgotten: If we are to build AGI, we are going to need to learn something from humans, how they reason and understand the physical world, and how they represent and acquire language and complex concepts.”

  62. Assessment: UnresolvedSaid

    “Systems like DALL-E 2, GPT-3, Flamingo, and Gato appear to be mind-blowing, but nobody who has looked at them carefully would confuse them for human intelligence.”

  63. Assessment: SupportedSaid

    “Success on a few problems with a particular strategy in no way guarantees that we can solve all problems in a similar way. It is sheer inductive folly to imagine otherwise, particular when the failure modes (unreliability, bizarre errors, failures in compositionality and discomprehension) have not changed since Fodor and Pinker pointed them out (in separate articles) in 1988.”

    What happened: Tests involving SCAN and RobustFill found continuing difficulty combining learned components in new ways, even with modified Transformers. This supports persistence of compositional generalization problems while explicitly acknowledging progress. (arXiv; Kensen Shi and colleagues, )

  64. Assessment: UnresolvedSaid

    “Virtually all of the world’s software, from web browsers to spreadsheets to word processors, still depends on knowledge engineering, and Sutton sells that short.”

  65. Assessment: UnresolvedSaid

    “So far, in most domains (not all, to be sure) Big Data has triumphed (for now) over careful knowledge engineering.”

  66. Assessment: UnresolvedSaid

    “As I have written, I am fine with thinking of Gato as an “Alt Intelligence”— an interesting exploration in alternative ways to build intelligence—but we need to take it in context: it doesn’t work like the brain, it doesn’t learn like a child, it doesn’t understand language, it doesn’t align with human values, and it can’t be trusted with mission-critical tasks.”

  67. Assessment: SupportedSaid

    “DeepMind’s newest star, just unveiled, Gato, is capable of cross-modal feats never seen before in AI, but still, when you look in the fine print, remains stuck in the same land of unreliability, moments of brilliance coupled with absolute discomprehension:”

    What happened: The launch paper reports one model spanning dialogue, captions, games and real robot control, including human-level or better performance on 23 Atari games, while acknowledging frequent poor dialogue responses. Its prior-work comparison supports novelty in that combined coverage, rather than novelty of multimodality itself. (arXiv; DeepMind, )

  68. Assessment: UnresolvedSaid

    “Right now, the predominant strand of work within Alt Intelligence is the idea of scaling.”

  69. Assessment: SupportedSaid

    “But it’s also pretty prone to surprising stumbles, like this uncomprehending reply to “a red cube on top of a blue cube”.”

    What happened: The authors discuss the exact red-cube/blue-cube prompt in Figure 14 and report recurring object–attribute mix-ups in reconstructions in Figure 15. This supports a recurring compositional weakness in the original model, without establishing a population-wide error rate. (arXiv; OpenAI, )

  70. Assessment: SupportedSaid

    “The history missed is the the fact that many neural net enthusiasts, have argued against symbols for decades, and the reality that is missed is the above-documented fact that symbolic descriptions like “red cube on blue cube” still elude the state-of-the-art in 2022.)”

    What happened: This dated scholarly account documents an explicit anti-symbolic strand well before the post. It is historical synthesis rather than a primary statement by an advocate. (Stanford Encyclopedia of Philosophy, )