Pundit ClaimsAI claims by Gary Marcus and Ed Zitron, with the evidence

Claims checked against evidence by Gary Marcus, page 15 of 19

Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.

Claims checked against evidence by Gary Marcus (1870)

Each of these claims was checked against dated evidence.

Page 15 of 19: claims 1401 to 1500 of 1870, ordered by the date first said, newest first. To find a claim by its words, use search.

  1. Assessment: UnresolvedSaid

    “Certainly no commercial product could ever work at a profit if you needed remote operators anything like that often.”

  2. Assessment: SupportedSaid

    “Nobody knows for SURE that there are no serious possible consequences of open source AI, e.g in terms of bioweapons, misinformation generation etc, as Séb Krier at DeepMind nicely highlights above.”

    What happened: The researchers explicitly describe unresolved protection against misuse and report experiments undoing safety training in publicly available Llama 2 models. This supports the absence of an established safety guarantee, without proving that serious real-world harm occurred. (arXiv; Simon Lermen, Charlie Rogers-Smith and Jeffrey Ladish, )

  3. Assessment: UnresolvedSaid

    “How effective this actually becomes depends a lot on the exact wording, and how things are enforced, and how much of it is binding as opposed to merely voluntary.”

  4. Assessment: UnresolvedSaid

    “Companies will poke for loopholes and surely find them.”

  5. Assessment: UnresolvedSaid

    “We will eventually see search engines that return neatly written paragraphs, and eventually see AI agents that can organize lives. Or at least our grandchildren or their children will.”

  6. Assessment: UnresolvedSaid

    “To get there, though, what we need is a much smarter AI, a kind that can reason, and not hallucinate, grounded in reality and not just corpus statistics— which require fundamental research of a sort that is currently getting starved out in the LLM euphoria.”

  7. Assessment: SupportedSaid

    “I—truly a techno-optimist, popular perceptions to the contrary—do think driverless will come, eventually.”

    What happened: The disclosure reports more than 250,000 paid weekly trips across Phoenix, San Francisco, Los Angeles, and Austin, with over 1,500 commercial vehicles. This establishes recurring practical deployment well beyond a demonstration. (Waymo, )

  8. Assessment: SupportedSaid

    “you don’t have to be superintelligent to create serious problems.The real issue is control.”

    What happened: Establishes actual fatal harm involving a specialized automated-driving system. Classifying this system as below superintelligence is an evaluator inference. (National Transportation Safety Board, )

  9. Assessment: SupportedSaid

    “We have no way of guaranteeing that large language models will reliably stick to facts, a point so thoroughly established by now I don’t think it needs a reference.”

    What happened: The contemporary survey considers factuality evaluation and enhancement for standalone and retrieval-augmented models, and describes fully factual LLMs as an outstanding research goal. This supports the ordinary research-state reading, although it does not prove a universal impossibility. (arXiv; Cunxiang Wang and colleagues, )

  10. Assessment: UnresolvedSaid

    “Current AI has no way to reliably anticipate consequences of actions that it might recommend, Asimov’s First Law of Robotics called for robots to do no harm, but current AI has no reliable way to anticipate harm.”

  11. Assessment: SupportedSaid

    “We have no way of guaranteeing that large language models won’t be jailbroken into saying unethical things. To the contrary, a just-published technique shows that current LLMs can be jailbroken quickly, and automatically.”

    What happened: The published demonstration reports roughly five-minute successful jailbreak generation and success rates of 60% for GPT-3.5, 62% for GPT-4, and 72% for PaLM-2. Success was substantially lower for Llama-2 and Claude, so the evidence establishes capability across multiple models rather than uniform vulnerability. (arXiv; Patrick Chao and colleagues, )

  12. Assessment: SupportedSaid

    “But, crucially, it often gets these wrong, particularly when they involve spatial reasoning beyond a cookie-cutter formula or reasoning about sequences of disparate events.”

    What happened: Both systems failed the pendulum and chemical-reaction problems. The spatial eclipse subset contained six wrong answers out of eight for GPT4+WA and five out of eight for GPT4+CI. These are repeated failures in the described collection. (New York University; Ernest Davis and Scott Aaronson, )

  13. Assessment: UnresolvedSaid

    “In a few cases GPT-4 set up the calculations correctly and the plug-in returned the correct answer, but then GPT-4 either misunderstood or refused to accept that answer.”

  14. Assessment: SupportedSaid

    “Second, GPT-4 doesn’t reliably understand how to use the tools.”

    What happened: By completion of testing, the report documented repeated unsuccessful Wolfram queries, failure to request a usable numerical representation, and Code Interpreter bypasses that produced calculation errors. These distinguish tool-use failures from merely misunderstanding a science problem. (Ernest Davis and Scott Aaronson, hosted by New York University, )

  15. Assessment: MixedSaid

    “In many cases, a naive user might take the results, invariably presented as fact, as correct, when they are not.”

    What held: The answer key for problem 3 is F, while the output confidently selects H. Problems 1 and 4–7 also contain incorrect declarative answers, supporting the recurring confident-error component. (New York University; Ernest Davis and Scott Aaronson, )

    What did not: The life-in-the-universe calculation explicitly warns about uncertainty in its estimates and repeats that qualification after reporting percentages. This contradicts the absolute assertion that results invariably appear as unqualified facts. (New York University; Scott Aaronson and Ernest Davis, )

  16. Assessment: SupportedSaid

    “Finding a way to get LLMs to reliably use external tools is far from trivial.”

    What happened: The July benchmark reports low completion rates for agents operating realistic websites with supporting tools, despite incorporating reasoning before acting. This supports the inference that connecting models to an actionable environment did not itself deliver dependable performance. (arXiv; Shuyan Zhou and colleagues, )

  17. Assessment: UnresolvedSaid

    “Thus far we have seen consistently unreliable results, from math to fact-checking.”

  18. Assessment: SupportedSaid

    “Certainly, adding these plug-ins does improve things.”

    What happened: On the common 120-problem subset, Figure 4a reports 48.33% success for ChatGPT with Code Interpreter, 45.0% with Wolfram Alpha, and 30.0% for vanilla GPT-4. The commercial products were not evaluated on the entire MATH test set. (arXiv; Qingyun Wu and colleagues, )

  19. Assessment: SupportedSaid

    “There is no question that the two plug-ins significantly extend GPT-4’s ability to solve math and science problems.”

    What happened: On the matched 120-problem subset, figure 4(a) reports 48.33% success for ChatGPT with Code Interpreter, 45.0% with Wolfram Alpha, and 30.0% for vanilla GPT-4. These are substantial practical gains for both plug-ins, although the comparison does not fully isolate differences between the web products and baseline configuration. (Qingyun Wu and colleagues, arXiv, )

  20. Assessment: UnresolvedSaid

    “First, neither GPT-4 nor the plug-ins have the commonsense world knowledge and reasonable abilities needed to reliably translate a word problem into a mathematical calculation.”

  21. Assessment: SupportedSaid

    “GPT-4 and similar current language-based AI programs notoriously struggle with mathematical calculations.”

    What happened: Table 3 reports exact-answer accuracy of 18.84% for GPT-4 and 10.00% for ChatGPT on 9,592 arithmetic cases. These substantial error rates support the ordinary reading of calculation difficulty. (arXiv; Zhen Yang and colleagues, )

  22. Assessment: SupportedSaid

    “On some problems that capable students might well find hard, the AI did perfectly; on other problems that lot of mathematically-inclined college students people would find easy, the AI failed miserably.”

    What happened: The report identifies correct high-dimensional probability and finite-field linear algebra answers alongside elementary failures. Their mathematical prerequisites support the qualitative difficulty contrast, but the authors explicitly report no human-subject testing. (New York University; Ernest Davis and Scott Aaronson, )

  23. Assessment: UnresolvedSaid

    “To our knowledge, no LLM-based system can yet reliably do what SHRLDU tried to do then:”

  24. Assessment: SupportedSaid

    “As is by now widely known, they are a hallmark of systems that generalize based on word distribution.”

    What happened: The survey identifies hallucination as a substantial reliability problem and organizes existing detection, explanation and mitigation research. This supports broad recognition of a recurring failure, without supplying a universal frequency. (arXiv; Yue Zhang and colleagues, )

  25. Assessment: UnresolvedSaid

    “And no matter what investors and CEOs might tell you, hallucinations will remain inevitable.”

  26. Assessment: SupportedSaid

    “To be sure, large language models are, in a lots of ways, an incredible advance over the AI of the 1970s. They can engage in a much broader range of tasks, and generally they need less hand-tinkering (Though they still need some – that’s what reinforcement learning with human feedback is really about.)”

    What happened: The report describes instruction-based generalization and evaluations across diverse held-out tasks. Compared with the bounded historical setting, this supports broader reuse with less task-specific programming, while its training-method discussion still documents substantial human design work. (Google Research, )

  27. Assessment: UnresolvedSaid

    “But in other ways, large language models are a profound step backwards; they lack stable models of the world, which means that they are poor at planning, and unable to reason reliably; everything becomes hit or miss..”

  28. Assessment: UnresolvedSaid

    “ChatGPT went utterly wild when Ernie pushed on the last one, omitting some that are there and and confabulating a standalone island in the middle of the kitchen that simply isn’t there:”

  29. Assessment: SupportedSaid

    “As we can see, multimodal training is not magically making hallucinations go away.”

    What happened: The release assessment reports hallucinations in GPT-4V itself. Its Be My Eyes discussion also reports reductions in hallucination frequency and severity during testing, supporting improvement without elimination. (OpenAI, )

  30. Assessment: UnresolvedSaid

    “The LLM never induces such an algorithm.”

  31. Assessment: UnresolvedSaid

    “The first author’s nine-year-old can do five digit multiplication better than an LLM.”

  32. Assessment: UnresolvedSaid

    “There are some applications where the current systems are reliable enough to be practically useful, though not nearly as many as is often claimed.”

  33. Assessment: SupportedSaid

    “Transformers and generative AI are getting a lot of attention, and have considerable utility, but they are nothing like the all purpose tools some people seems to imagine them to be.”

    What happened: The cumulative usage disclosed on this date demonstrates substantial engagement. The same paragraph documents uses including illustrated stories, social-media thumbnails and design inspiration, supporting practical utility without establishing universal reliability. (Microsoft Bing, )

  34. Assessment: SupportedSaid

    “No doubt the next generation of systems will have greater abilities and more extensive applications and will somewhat mitigate the weaknesses.”

    What happened: The first incremental successor reports fewer false statements than the October-generation Claude 2.0 on the company's factual-question evaluation. This directly supports mitigation of an existing weakness, within that evaluation's scope. (Anthropic, )

  35. Assessment: UnresolvedSaid

    “About 18 months have passed; although there are certainly improvements (the images are vastly better), owing presumably in large part to a larger training set, the fundamental approach (mapping words to images, rather than constructing intermediate models of the world and how language and images relate to those models) remains the same, and so the core issue remains.”

  36. Assessment: UnresolvedSaid

    “It’s the same issue of canonical versus noncanonical statistics, all over again.”

  37. Assessment: UnresolvedSaid

    “though I agree with Geoff that eventually AI will have a big impact on drug design. Perhaps with current techniques, perhaps not; we will have to see.”

  38. Assessment: UnresolvedSaid

    “I think it’s fair to say that generative AI works better than most people expected.”

  39. Assessment: MixedSaid

    “It makes current AI hard to debug (nobody knows for example how to ground them in facts), and it makes them difficult predict, which means, unlike calculators or spreadsheets, we don’t really know what’s going to happen when we ask them a question.”

    What held: The opening paragraph explicitly connects poor understanding with difficulty diagnosing failures, fixing them, and certifying safety. The research also demonstrates limited progress in interpreting and steering a small model, so it supports difficulty rather than total ignorance. (Anthropic, )

    What did not: The original paper describes retrieval and individual verification of proposed facts against Wikipedia, with improved factual accuracy. This contradicts the absence of grounding methods, but does not establish error-free generation. (arXiv; Sina J. Semnani and colleagues, )

  40. Assessment: SupportedSaid

    “Just like we can’t guarantee that they won’t make stuff up, we don’t know how to guarantee that they won’t write flawed code.”

    What happened: The developer explicitly acknowledges factual fabrication in its contemporary model. The surrounding discussion recommends review and grounding appropriate to the application rather than claiming assured factual correctness. (OpenAI, via arXiv, )

  41. Assessment: UnresolvedSaid

    “And emphasize that wholesale, automated fake news will be used both to manipulate markets and elections, and might undermine democracy.”

  42. Assessment: UnresolvedSaid

    “Your best example of a spectacular machine invention is in fact a perfect example of the kind of parroting and theft of intellectual property that is characteristic of large language models.”

  43. Assessment: SupportedSaid

    “When it comes to AI in particular, we are getting way ahead of our skis, rushing forward a technology we don’t fully understand.”

    What happened: This documents actual expansion of access to GPT-4. It establishes deployment growth, although not that the chosen pace was excessive. (OpenAI, )

  44. Assessment: SupportedSaid

    “There’s some information in there, but whatever’s there is often both patchy and problematic.”

    What happened: Human evaluation of generated biographies found substantial mixtures of supported and unsupported atomic facts. This supports both useful factual content and recurring reliability problems within that task. (arXiv, )

  45. Assessment: SupportedSaid

    “The connections in chatbots are very different from the connections in the brain; it’s a mistake to compare them directly in this way.”

    What happened: The review documents materially different learning mechanisms. The evaluator's inference is that counting connections alone does not control for those differences and therefore cannot establish superior learning per connection. (arXiv, )

  46. Assessment: SupportedSaid

    “Actually … a lot of us still think you are declaring victory prematurely.”

    What happened: Mitchell and Krakauer describe competing research camps and unresolved questions about understanding. This supports the existence of substantive disagreement, rather than unanimous acceptance of a completed victory. (arXiv, )

  47. Assessment: UnresolvedSaid

    “You know how many radiologists have been replaced by machines seven years later? Zero.”

  48. Assessment: UnresolvedSaid

    “I can’t either; there’s a lot we can do to help, but nothing I can see either to absolutely guarantee safety. Rushing ahead is creating risk.”

  49. Assessment: UnresolvedSaid

    “There is no internal model of what she actually did, and hence no capacity to support reliable inference, hence no guarantee that anything it writes about her will be true.”

  50. Assessment: UnresolvedSaid

    “Without flexible underlying models that can reliably support inferences on novel questions, we can expect more of the same of what we are seeing over and over now: more hallucinations, more reasoning that is hit or miss, and nothing like the stability we should insist on for future AI.”

  51. Assessment: UnresolvedSaid

    “It’s the same problem: correlations are fragile, and you can’t count on them.”

  52. Assessment: SupportedSaid

    “It’s also not even remotely new.”

    What happened: Louwerse and Zwaan recovered geographic relationships among 50 US cities using newspaper text, latent semantic analysis and multidimensional scaling. This directly establishes the earlier finding Marcus describes. (Cognitive Science; abstract reproduced by PubMed, )

  53. Assessment: UnresolvedSaid

    “By my count, though I acknowledge I may have missed one, there are at least 7 falsehoods.”

  54. Assessment: SupportedSaid

    “Rumors that LLMs have solved then hallucination problem are greatly exaggerated.”

    What happened: The revised paper identifies hallucination as an unresolved contemporary problem and reports experiments reducing it through verification. The event date identifies this current-state assessment, not the undisclosed dates of individual experimental runs. (arXiv; Shehzaad Dhuliawala and colleagues, )

  55. Assessment: UnresolvedSaid

    “This all happens not because LLMs are smart, in the sense of being able to compare what they say with the world, but simply because they are good at making new sentences that sound old sentences, the essence of autocomplete.”

  56. Assessment: UnresolvedSaid

    “If OpenAI were obliged to retrain models only on materials for which they had consent, it significantly would weaken their results, and cost a great deal of money to do that retraining.”

  57. Assessment: UnresolvedSaid

    “There’s not a huge moat here to protect OpenAI from competitors. The central technology at OpenAI (as far as anyone knows) is large language models, and many other companies know how to build them.”

  58. Assessment: SupportedSaid

    “Explicit verbal instruction is key in training human employees, and LLMs aren’t very good at following those instructions.”

    What happened: PaLM 2 S satisfied every checked instruction on 43.07% of prompts under strict scoring and 46.95% under relaxed scoring; individual-instruction accuracy was 55.76% and 59.11%, respectively. These August results show frequent explicit-constraint failures, but this smaller model alone cannot establish performance across all leading LLMs. (arXiv; Jeffrey Zhou and colleagues, )

  59. Assessment: SupportedSaid

    “Hallucinations are a huge, unsolved issue.”

    What happened: OpenAI identifies continuing factual unreliability and recommends safeguards or avoiding some high-stakes uses. The same section reports substantial improvement over preceding models, which limits any reading that hallucination mitigation had made no progress. (OpenAI, )

  60. Assessment: SupportedSaid

    “The competitors may not (yet) be as good, but they are free, which will suit some customers fine.”

    What happened: The case study documents a deployed customer application and reports positive reception and further integration. Together with the release terms, this establishes that a free competing model suited a customer. (Meta, )

  61. Assessment: UnresolvedSaid

    “This symmetry fail is mighty big, a mighty persistent error that has endured for decades.”

  62. Assessment: SupportedSaid

    “On made-up facts, in a first experiment, the model was at zero percent correct, on celebrities, in a second experiment, performance was still dismal.”

    What happened: Table 1 gives GPT-3-175B synthetic reverse accuracies of 0.0% and 0.1% for the two directions. Celebrity reversal succeeds in 33% of cases, counting a correct answer in any of ten attempts, versus 79% for identifying parents. (arXiv; Berglund et al., )

  63. Assessment: SupportedSaid

    “Forty one years later, neural networks (at least of the popular variety) still struggle with this. They still remain pointillistic masses of blurry memory, never as systematic as reasoning machines ought to be.”

    What happened: The study evaluates GPT-3, ChatGPT and GPT-4 on multiplication, relational logic puzzles and dynamic programming. Performance deteriorates sharply as compositional complexity increases, supporting the claim that popular models struggle with systematic reasoning. (arXiv; Dziri et al., )

  64. Assessment: SupportedSaid

    “The shocking (yet, in historical context, see below, unsurprising) answer is no:”

    What happened: Table 1 reports strong directional asymmetry, and section 2.1.2 finds no detectable advantage for the correct reversed name over random names. Section 1 explicitly excludes in-context learning from this failure. (arXiv; Berglund et al., )

  65. Assessment: UnresolvedSaid

    “It’s just a matter before people start to realize that we need some genuinely new ideas in the field, either new mechanisms (perhaps neurosymbolic2), or different approaches altogether.”

  66. Assessment: SupportedSaid

    “Voluntary guidelines aren’t laws, and they aren’t enforcement mechanisms, either; companies can still do as they please.”

    What happened: The instrument expressly distinguishes voluntary commitments from future regulations. Reading the complete document reveals no new statutory authority, compulsory enforcement process or penalties for failing to implement its eight safeguards. (White House, )

  67. Assessment: UnresolvedSaid

    “If we don’t know what is in the data sets, we can’t know what biases these systems will spread through society if they are widely used, we can’t know the extent to which individual content creatore are being exploited, and worse any outside efforts to mitigate risks are greatly undermined.”

  68. Assessment: SupportedSaid

    “At least in the United States, policy around AI is still largely notional, not actual.”

    What happened: The administration described major executive and legislative safeguards as work still to come. This directly supports the contextual reading of the claim. (White House, )

  69. Assessment: UnresolvedSaid

    “The apparent economics of AI (we will see if they yet come to pass as imagined) are too seductive to too many, to people building AI companies, to people investing in AI companies, to people getting paid 6 and even 7 digit salaries, and to people expecting campaign donations from big tech companies.”

  70. Assessment: UnresolvedSaid

    “We are obviously never going to have a moratorium anytime in the near-term.”

  71. Assessment: UnresolvedSaid

    “We have vastly more attention on AI policy, throughout the world, and I definitely give the pause letter (which I signed but didn’t write) some credit for that.”

  72. Assessment: SupportedSaid

    “Large language models still make stuff up (“hallucinate”), still can’t be trusted in their reasoning, and remain unreliable.”

    What happened: Human fact-checking found substantial unsupported content in generated biographies. These are factual-precision measurements against the study's knowledge source, not general error rates for every ChatGPT answer. (Sewon Min and colleagues, via arXiv, )

  73. Assessment: SupportedSaid

    “Your private data are going to feed the insatiable beast, too — like it or not.”

    What happened: Google describes reviewers assessing users' nonpublic conversations and supplying improvements used by its models. The same FAQ explicitly includes generative-model training among uses of saved activity when Keep Activity is enabled; e2 establishes the default-participation policy. (Google, )

  74. Assessment: SupportedSaid

    “And even if this specific example is patched up, there will inevitably be others of a similar flavor.”

    What happened: Google identifies AI Overviews as powered by a customized language model and acknowledges advice about putting glue on pizza, alongside mishandling a question about eating rocks. These are subsequent commonsense failures matching the surrounding description of plausible answers containing nonsense. (Google, )

  75. Assessment: UnresolvedSaid

    “That has meant, for example, no tangible progress on getting the big companies to commit to revealing what data their models were trained on; we have transparency in name, but not in deed.”

  76. Assessment: SupportedSaid

    “Indeed, what we have seen on the AI front so far in the US, for the most part, is voluntary guidelines – stuff that the companies are comfortable with.”

    What happened: The announcement directly establishes that the participating companies accepted voluntary commitments. Acceptance supports the ordinary reading of company comfort, without proving their private preferences. (The American Presidency Project, reproducing White House remarks, )

  77. Assessment: UnresolvedSaid

    “The default right now is regulatory capture: the companies will make the rules, telling the governments what they are and are not comfortable with. The government will go along.”

  78. Assessment: UnresolvedSaid

    “Without data transparency, we can’t make sure our models are without bias, or even know the sources of the biases they hide within, we can’t truly understand the scope and limits of their generalization (which is central to understanding what they might do in unexpected circumstances) and we are hamstrung in our ability to mitigate their harmful consequences: what is unknown is hard to fix.”

  79. Assessment: SupportedSaid

    “When I think about how poor his company (Meta) is even at making LLMs that don’t hallucinate, I get worried.”

    What happened: The 70B chat model achieved 64.14% generations judged both truthful and informative on TruthfulQA, showing a substantial gap from reliable factual answering. This result is also present in the dated July technical report; unsuccessful answers cannot all be counted as hallucinations. (Meta, )

  80. Assessment: UnresolvedSaid

    “But if and when we get there, will they be dominating? Non-dominating? The honest answer is we haven’t a clue.”

  81. Assessment: UnresolvedSaid

    “We aren’t going to be able calculate p(doom) to three decimal places.”

  82. Assessment: UnresolvedSaid

    “Yet a really large fraction of the AI and AI-adjacent community doesn’t really want to talk about any of this at all.”

  83. Assessment: SupportedSaid

    “this too seems like a rather large assumption that is far from proven, a possibility but not demonstrably a default,”

    What happened: The authors explicitly limit the inference from their formal results to hypothetical superintelligent agents, noting differences between optimal and learned policies. This supports an evidential gap in a key step toward default extinction, although it does not by itself evaluate every possible extinction argument. (arXiv; Alexander Matt Turner and coauthors, )

  84. Assessment: UnresolvedSaid

    “Some of the specific arguments Yudkowsky and Bostrom have made may well be weak, but there is no solid argument that the number is zero, or even that the risk is infinitesimal.”

  85. Assessment: SupportedSaid

    “And of course not every enterprise-scale experiment will be successful.”

    What happened: Reporting from Tuesday's bankruptcy hearing documents inability to fulfill the district contract and preparations to close the supplier. Together with the deployed chatbot's shutdown, this establishes failure of the original implementation, beyond a routine temporary interruption. (The 74, )

  86. Assessment: SupportedSaid

    “The backlash as begun.”

    What happened: Gioia's essay is direct evidence that prominent critical commentary existed before Marcus's post. Its claims about demand are not accepted as evidence of actual demand. (The Honest Broker / Ted Gioia, )

  87. Assessment: UnresolvedSaid

    “In less than a year, ChatGPT has gone from being mistaken for AGI to being the butt of a joke, and an insulting shorthand for robotic, incoherent, unreliable, and untrustworthy.”

  88. Assessment: UnresolvedSaid

    “And the financial challenges are starting to mount up.”

  89. Assessment: SupportedSaid

    “What would make artificial general intelligence general is generality: the ability to cope with new things. And that’s precisely where current AI is still lacking.”

    What happened: ConceptARC tested generalization across variations of concepts. Humans substantially exceeded specialized solvers; GPT-4 scored below 30% on 15 of 16 concepts, supporting a substantial limitation under these test conditions. (arXiv; Arseny Moskvichev, Victor Vikram Odouard and Melanie Mitchell, )

  90. Assessment: UnresolvedSaid

    “This is even more true four years later, now that we have systems that ingest massive amounts of data from both public and commercial sources, quite possibly including eg sample law school and medical exams etc.”

  91. Assessment: SupportedSaid

    “But it is also well-known throughout the industry that benchmarks are often quite gameable.”

    What happened: The review documents benchmark shortcuts across vision, language, reinforcement learning and algorithmic decision-making. This supports a recurring, publicly recognized problem rather than an isolated vulnerability. (arXiv, )

  92. Assessment: UnresolvedSaid

    “We are not even close to machines that exceed the capacity of typical humans to cope with unusual circumstances while driving”

  93. Assessment: UnresolvedSaid

    “Without a sense of what it is in the training corpus, it’s almost impossible to conclude anything about what a machine truly understands and how general that is.”

  94. Assessment: SupportedSaid

    “Basically, beating benchmarks doesn’t tell you as much as you might think. Often when you do, it’s a statistical artifact.”

    What happened: Models scoring well on MNLI failed controlled examples that invalidated their syntactic shortcuts. The experiment links apparent benchmark competence to heuristics that do not reliably perform the intended inference. (arXiv, )

  95. Assessment: SupportedSaid

    “Machines often solve them for the wrong reasons. Building them is an unsolved problem within AI.”

    What happened: The 2023 version describes benchmark design as an ongoing difficulty and reports that single-sentence label leakage persists despite mitigation efforts. The event date denotes that research-state assessment, not the date of every underlying experiment. (arXiv; Michael Saxon and colleagues, )

  96. Assessment: SupportedSaid

    “Humans can learn rules of games like chess from modest amounts of explicit instruction; LLMs can’t stick to explicit rules.”

    What happened: The accompanying demonstrations also document illegal GPT-3.5 moves with specialized prompting and a GPT-4 failure to obey the instruction to output only its own move. The same report includes successful legal games, so it supports unreliable adherence rather than total inability. (Adam Yedidia, LessWrong, )

  97. Assessment: UnresolvedSaid

    “Humans can write summaries of things without hallucination; LLMs cannot do so reliably.”

  98. Assessment: UnresolvedSaid

    “18 months ago, before the LLM euphoria, we all took for granted that AGI wasn’t just about beating ordinary humans.”

  99. Assessment: UnresolvedSaid

    “in my view it’s a subtle redefinition that dramatically moves the goal posts.”

  100. Assessment: SupportedSaid

    “The dirty little secret that pretty much everyone in the AI community knows is that doing well on a benchmark is not actually tantamount to being intelligent.”

    What happened: CheckList identifies basic linguistic failures in commercial and leading models across three tasks despite strong benchmark results. These empirical failures support the inference that benchmark success alone does not establish broad, reliable competence. (arXiv, )