Checked claims, page 35 of 36
Claims by Gary Marcus and Ed Zitron, as of 2026-11-11.
Checked claims (3,554)
Claims we checked against evidence.
Claims 3,401 to 3,500 of 3,554, newest first. Page 35 of 36.
Assessment: SupportedSaid
“LLMs are unruly beasts; nobody knows how to make them refrain 100% of time from insulting users, giving bad advice, or just plain making stuff up.”
What supported it: OpenAI described reliable refusal of unsafe instructions as unsolved despite improvements from human-feedback training. Its limitations section also reports remaining toxic outputs and fabricated facts. (OpenAI, )
Assessment: ContradictedSaid
“It's worth asking since (a) Amazon is apparently greatly reducing Alexa's staff, (b) hasn't added significant features in years, and (c) seems clumsy and limited compared to GPT-3.”
What contradicted it: The dated launch account describes multiparty interaction and speech directed at Alexa without repeatedly saying the wake word. This is a substantive interaction feature within the claimed period, although it does not establish GPT-3-like open conversation. (Amazon Science, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Pitch perfect and utterly bogus imitations of science and math, presented as the real thing.”
What supported it: The preserved response presents a purported paradox with formal sections and a solution, treating the sum of two 0.5 terms as 0.4 to reach this conclusion. Direct arithmetic gives 1, so the mathematical exposition is invalid despite its factual presentation. (Ernest Davis and Andrew Sundstrom, NYU-hosted experiment record, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Knockoffs of GPT-3 are getting cheaper and cheaper, which means that the cost of generating misinformation is going to zero, and the quantity of misinformation is going to rise—probably exponentially.”
What supported it: The release compares optimized and baseline implementations of open text generators, including BLOOM and OPT, using dollars per million generated tokens. Its cost-sensitive benchmarks report substantial reductions, directly supporting cheaper attainable inference before the post. (Microsoft DeepSpeed, )
Assessment: SupportedSaid
“Knockoffs of GPT-3 are getting cheaper and cheaper, which means that the cost of generating misinformation is going to zero, and the quantity of misinformation is going to rise—probably exponentially.”
What supported it: The experiments establish that alternatives can produce convincing election-disinformation content. For ten articles daily over a month, the cost section estimates $0.30 using Gemini or $9 using hosted Llama 3 70B, compared with $4,500 for the cited human-writing service; these are generation-stage estimates, not total campaign budgets. (PLOS ONE, )
Assessment: SupportedSaid
“Scaling here was largely AWOL; even the best model was only at 80.6%, and for most models, scaling had at best a neglible effect.”
What supported it: The best overall mean is 80.6% with 30 examples; Section 4.1 reports near-zero or decreasing slopes for many families, with only Cohere, T0 and InstructGPT-3 performing best at their largest tested size. The same source qualifies the headline: structured prompts average 81.7%, and Table 58 includes an individual-template score of 82.8%. (arXiv; Laura Ruis and colleagues, )
Assessment: SupportedSaid
“Scaling is an empirical observation, not a guaranteed, continued law of nature.”
What supported it: The foundational paper presents empirical relationships and explicitly qualifies their theoretical status. Appendix C and section 6.3 discuss limits to trusting or extending those relationships, supporting the absence of an indefinite guarantee. (Jared Kaplan and colleagues, arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“having a machine look at a radiology scan is very different from replacing the entire job of a radiologist”
What supported it: The society's contemporary statement describes patient advocacy, safety, treatment and communication responsibilities alongside diagnosis. These substantive duties support the distinction between interpreting images and replacing the whole clinical role. (European Society of Radiology; Insights into Imaging, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“It’s way too much charity to deep learning to claim that there is any kind of direct link.”
What supported it: The author's account describes mapping, correlations and effects of changing model depth and attention. Inferring that these do not establish identical biological mechanisms is the evaluator's assessment, not an explicit concession in this quotation. (Charlotte Caucheteux, reproduced by Thread Reader, )
Assessment: SupportedSaid
“So if you have a correlation of R = .5, you actually “explain” (really, just “predict”) only 25% of the variance—which means fully three-quarters (not half) of the variability remains unexplained.”
What supported it: The paragraph ties this statement to R = 0.50. Squaring that coefficient yields 25% accounted for and 75% remaining. (Scientific Reports, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Of course we see the same thing over and over in language systems, too.”
What supported it: The surrounding results describe recurrent false-premise acceptance, excessive hedging, and constraint failures despite strong language-task performance. These directly match the behavioral pattern Marcus describes. (arXiv; OpenAI research authors, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“There was a lot of bluster (we will 100x the world’s productivity) but no road map.”
What contradicted it: Tesla supplied tentative delivery timing. The same transcript outlines prototype improvement followed by factory deployment and progressively broader uses. (Tesla, transcribed by Stock Analysis, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Sooner or later he will recognize that humanoid robots, too, pose a whole set of challenges that lie well beyond our current grasp.”
What supported it: Google's robotics research head describes unresolved real-world adaptation and understanding, with helpful robots still limited to restricted kitchen tasks and explicit safety controls. Applying these shared robotics limitations to general household humanoids is an inference, rather than a direct humanoid safety test. (Google, )
Assessment: SupportedSaid
“The real problem w robots has never been demos, anyway (there have been many, for decades); it’s making robots in robust in the real world.”
What supported it: SRI dates Shakey's development to 1966–1972 and describes integrated perception, reasoning, navigation, and task execution. The endpoint records demonstrated robotic capabilities roughly five decades before the post. (SRI International, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Over and over, the field of AI has discovered otherwise. Guy van den Broeck’s study is a perfect example of this.”
What supported it: Zhang and colleagues, including van den Broeck, found near-perfect performance within a training distribution but substantial failures under different sampling of the same logical problem space. Their analysis attributes this to statistical features rather than acquisition of the intended reasoning procedure. (arXiv, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Over and over I have stressed that you could have a good model of engineering without being a good model of humans, or vice versa.”
What supported it: Faggella's interview report describes Marcus contrasting evolved human memory with more reliable computer memory. It records an earlier example of engineering desirability differing from human fidelity; the quotation preserves the source's apparent typo. (Emerj; Dan Faggella, )
Assessment: SupportedSaid
“But the smart money is still on being cautious about this, and in recognizing that what the Facebook researchers said in 2019 is still the case.”
What supported it: Reducing misleading correlations improved independent-test performance for BERT and stronger RoBERTa and ALBERT models, supporting continued susceptibility in 2022. The same experiments found better generalization from larger models and debiasing, which limits any claim that no genuine progress occurred. (arXiv; ACL 2022 author manuscript, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“On #2, the answer is mixed. They do well on some inputs (like astronaut rides horse) but more poorly on others (like horse rides astronaut, which I discussed in an earlier post).”
What supported it: The study's human judgments show some successful matching alongside widespread relational failures. Its results were reported by July; the precise image-generation dates are not supplied. (arXiv; Colin Conwell and Tomer Ullman, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“The dirty secret in current AI is that for all the bloviating about how current systems have mastered human language, they are still really weak on compositionality, which most linguists (since Frege, over a century ago) would agree is at the very core of how language works.”
What supported it: Winoground found large human-model gaps across several vision-language model families on image-caption matching that depends on composition. The finding supports a broad weakness within the tested contemporary systems. (arXiv; Thrush and colleagues, )
Assessment: SupportedSaid
“The progress that Alexander is vaunting shows that the system did better on a handful of items than DALL-E2 (which was not even state of the art at compositionality when it appeared), but not that the newer system (Imagen) systematically groks compositionality”
What supported it: The page specifies five prompts, reports DALL-E 2 at 0/5 and Imagen at 3/5 under Alexander's scoring, and contains no systematic retest of earlier successes. Alexander also expressly declines to claim full mastery, so the methodological limitation does not establish that he claimed otherwise. (Astral Codex Ten, )
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“we were right to be skeptical; so far, four and half years later, Duplex hasn’t gotten a whole lot further.)”
What contradicted it: Google reported over a million completed bookings and over three million business-information updates across eight countries, alongside the quoted automation rate. These reported deployments show substantial operational progress beyond a limited demonstration. (Google, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“This should be a piece of cake, but for currently popular systems, it’s not.”
What supported it: The group score requires all relevant caption-image comparisons to be correct; every tested model scored below its random baseline. Some models exceeded chance on the less demanding text score, but remained far behind humans and did not reliably solve the complete matching task. (arXiv; Thrush and colleagues, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Driverless car demos have been around for decades, but making them work reliably has proven far harder than most people realized.”
What supported it: The report records a direct industry admission and describes executives narrowing expectations because of technical complexity. A readable primary transcript was not obtained, so contemporary reporting is used. (WIRED, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Performance hovered at around 75% in the best case, when the robots had a limited number of options to consider.”
What supported it: The reported overall execution rates are 74% in the mock kitchen and 60% in the real kitchen. Thus the better overall setting is approximately 75%; planning is a separate measure. (arXiv; Michael Ahn and colleagues, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“First, the language technology that the new system relies on is well-known to be problematic and second it is likely to be even more problematic in the context of robots.”
What supported it: PaLM's own announcement acknowledges an existing research literature on risks and describes analyses of bias and model outputs. This directly supports recognition of problems in the relevant technology. (Google Research, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“And we know from all the research on large language models that they simply aren’t reliable enough to give us anything like 100% clarity on user intentions.”
What supported it: The August evaluation identifies recurring problems with negation and ambiguous references, supporting the ordinary reading that interpretation remained unreliable. The paper also reports improvements from chain-of-thought prompting, so this does not establish an unavoidable limitation. (Ahn et al., Robotics at Google and Everyday Robots, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“it’s even less clear how it would work with words like no and not (a major sore spot for current neural networks).”
What supported it: The study reports substantial RoBERTa performance losses on subsets where negation matters: QNLI macro F1 is 0.67 for important negations versus 0.92 for unimportant negations. These are dataset-subset comparisons, not matched sentence pairs; the findings were already reported in the March preprint. (Association for Computational Linguistics, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Since then, the results on benchmarks have gotten better, but there’s still something missing.”
What supported it: PaLM's contemporary report documents continued benchmark improvements. This supports the improvement component, without establishing that benchmark success implies general understanding. (arXiv; Chowdhery et al., )
Assessment: ContradictedSaid
“(LLM’s like GPT produce well-formed prose but do not produce interpretable representations of utterances that reflect structured relationships between the parts of those sentences.)”
What contradicted it: The study uses late-2021 GPT-3 and Codex models to map utterances into structured meaning representations. It demonstrates a task capability, while its use of prompts and decoding machinery limits conclusions about ordinary internal representations. (arXiv; Richard Shin and Benjamin Van Durme, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“noncanonical reorderings such as these remain problematic for current AI approaches.)”
What supported it: Winoground requires matching two images to captions containing the same words in different orders. Its poor group scores across contemporary vision-language models support substantial difficulty preserving the changed relationships. (Thrush et al., CVPR 2022 / arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“Current systems simply do not postulate such cognitive models—and so, they technically can’t prove themselves wrong, because they do not even have a representation of what being “right” would look like.”
What contradicted it: The experiments evaluate models critiquing their own outputs against supplied passages or task constraints, and using critiques to improve answers. This contradicts the functional claim of complete inability, while leaving the nature of internal cognitive representations unsettled. (arXiv; William Saunders and colleagues, OpenAI, )
Assessment: ContradictedSaid
“The downstream consequence of all of this is that these models have no grasp whatsoever on truth.”
What contradicted it: GPT-judge classified the factual truth of answers and transferred across held-out answer-producing model families. This contradicts absolute behavioral incapacity, but all benchmark questions were represented in training, so it does not establish unrestricted factual understanding. (Association for Computational Linguistics; Stephanie Lin, Jacob Hilton and Owain Evans; ar5iv mirror, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Because it shows that the people who really claim to be working on full self-driving (which Tesla waffles on1) are having plenty of problems of their own.”
What supported it: The record confirms repeated ADS crash involvement in the initial reporting period. Its abstract defines reportability by ADS use within 30 seconds of a crash, not by a finding of ADS fault. (Transportation Research Board, summarizing NHTSA, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Level 5 has to handle it all; everyday stopped vehicles, weird outliers and so forth. We’re just not there.”
What supported it: The contemporaneous report directly addresses the claimed industry-wide state. Reputable reporting is used because no readable primary source establishing that complete industry-wide negative was obtained. (The Verge, )
Assessment: SupportedSaid
“Indeed, the NHTSA report data show that even some pretty ordinary circumstances, like avoiding stopped vehicles, are continuing to prove problematic.”
What supported it: The surrounding summary identifies these as additional Autopilot collisions with stopped first-responder vehicles. Its incident list dates Orlando to August 2021, Petaluma to September 2021, and Desert Center to January 2022; the event date records the latest of these in-window incidents. (National Highway Traffic Safety Administration, )
Assessment: UnresolvedSaid