Claims checked against evidence, page 28 of 31
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Claims checked against evidence (3100)
Each of these claims was checked against dated evidence.
Page 28 of 31: claims 2701 to 2800 of 3100, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Basically, beating benchmarks doesn’t tell you as much as you might think. Often when you do, it’s a statistical artifact.”
What happened: Models scoring well on MNLI failed controlled examples that invalidated their syntactic shortcuts. The experiment links apparent benchmark competence to heuristics that do not reliably perform the intended inference. (arXiv, )
Assessment: SupportedSaid
“Machines often solve them for the wrong reasons. Building them is an unsolved problem within AI.”
What happened: The 2023 version describes benchmark design as an ongoing difficulty and reports that single-sentence label leakage persists despite mitigation efforts. The event date denotes that research-state assessment, not the date of every underlying experiment. (arXiv; Michael Saxon and colleagues, )
Assessment: SupportedSaid
“Humans can learn rules of games like chess from modest amounts of explicit instruction; LLMs can’t stick to explicit rules.”
What happened: The accompanying demonstrations also document illegal GPT-3.5 moves with specialized prompting and a GPT-4 failure to obey the instruction to output only its own move. The same report includes successful legal games, so it supports unreliable adherence rather than total inability. (Adam Yedidia, LessWrong, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“The dirty little secret that pretty much everyone in the AI community knows is that doing well on a benchmark is not actually tantamount to being intelligent.”
What happened: CheckList identifies basic linguistic failures in commercial and leading models across three tasks despite strong benchmark results. These empirical failures support the inference that benchmark success alone does not establish broad, reliable competence. (arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“It highlights the inadequacy of the voluntary agreements such as the one that 7 tech companies recently made with The White House. Those agreements called for watermarking, but because they are voluntary, they do nothing to prevent bad actors from rolling their own automated misinformation software.”
What happened: The full document assigns undertakings to participating companies and contains no obligation on independent nonparticipants. It also includes safeguards against misuse and unauthorized access, which could provide indirect benefits without binding outside developers. (The White House, publishing participating companies' commitments, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Social media companies better watch out, too; they will be polluted in much the same way as the internet as a whole.”
What happened: The report documents multiple covert campaigns using generated material on Telegram, X, Facebook, Instagram, and websites, including Doppelganger and Zero Zeno. These establish repeated distribution through the mechanism described, while the report also says AI had not meaningfully increased audience reach. (OpenAI, )
Assessment: SupportedSaid
“And it’s on: the LLM-driven enshittification of the internet.”
What happened: NewsGuard identified 49 apparently AI-generated sites using published chatbot artifacts and content inspection, including a fabricated report of President Biden's death. Its report also says most identified sites did not advance misinformation, and some operators disputed its characterization. (NewsGuard, )
Assessment: SupportedSaid
“And of course LLMs are fully capable of make up truly terrible advice, too, like this disconcerting recipe from last week:”
What happened: The report dates Hehir's dangerous recipe to August 4 and identifies its LLM. It specifies chloramines and reports that attempted reproduction on August 10 was blocked. (Ars Technica, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“The underlying technology—often referred to as Foundation Models, because systems are fine-tuned on top of large pretrained (foundation) model—is deeply unstable and thus difficult to engineer into reliable products.”
What happened: Comparisons of March and June model versions document formatting failures that prevented code execution and caused a LangChain parser to fail, directly connecting model behavior to integration difficulty. The results also limit the claim: GPT-4's agent performance improved, and removing non-code wrappers increased its code-test success from 52% to 70%, so behavioral change did not imply declining underlying capability. (arXiv; Lingjiao Chen, Matei Zaharia, and James Zou, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“We have not one, but many, serious, unsolved problems at the core of generative AI — ranging from their tendency to confabulate (hallucinate) false information, to their inability to reliably interface with external tools like Wolfram Alpha, to the instability from month to month (which makes them poor candidates for engineering use in larger systems).”
What happened: OpenAI explicitly acknowledged continuing factual errors in its released model. The same section reports improvements, so persistence should not be confused with lack of progress. (OpenAI, )
Assessment: SupportedSaid
““Scaling” systems by making them larger has helped in some ways, but not others; we still really cannot guarantee that any given system will be honest, harmless, or helpful, rather than sycophantic, dishonest, toxic or biased.”
What happened: The study documents tasks where increasing scale worsened performance despite broader gains in prediction and other tasks. It also reports exceptions and trend reversals, so it does not establish that scaling generally fails. (Ian R. McKenzie and colleagues, arXiv, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“The technology we have now is built on autocompletion, not factuality.”
What happened: The report explicitly identifies predictive completion as GPT-4’s base objective. It separately reports factuality gains from post-training, supporting the foundation reading while qualifying a literal denial of factuality work. (OpenAI, arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“They will gladly replace artists, writers and any other role that they believe can be automated in a heartbeat.”
What happened: The report documents dismissal of the Spanish editorial staff and replacement with automated translations. It supplies a completed editorial substitution; no direct employer announcement was obtained. (Ars Technica, )
Assessment: UnresolvedSaid
Assessment: MixedSaid
“In just a few short months, the race to regulate AI, possibly at an international level, has gone from barely on the table to heated.”
What held: This primary government announcement describes summit planning, recent G7 discussions and meetings with frontier AI companies. It supports the high-level activity component by June, but does not establish a marginal starting point. (UK Prime Minister's Office, )
What did not: The February meeting was already the committee's fourth plenary, with negotiations and draft convention provisions under examination. That challenges the description of international AI regulation as barely on the table early in 2023. (Council of Europe, Committee on Artificial Intelligence, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“While AI will undoubtedly help us become more monetized and more interrupted by billion and trillion-dollar enterprises, it’s arguable that we haven’t already entered a digital society where the corporations are completely in control.”
What happened: The section confirms that ads directly within AI Overviews were available to US mobile users after testing, adding a placement inside the generated answer. That establishes AI-mediated commercialization and an additional commercial intrusion under the stated reading. (Google, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“AI might help profligate this kind of content, but you are completely delusional if you can’t see that the modern content mill of the internet is already full of crap specifically built to convince Google to give it more traffic.”
What happened: The investigation describes padded commercial copy, interviews a producer about writing for Google, and identifies mass SEO-oriented publishing at media outlets. This supports an established production practice; reputable reporting is used because no comprehensive contemporaneous primary measurement was found. (The Verge, )
Assessment: SupportedSaid
“We also know, for example, that Bing has defamed people, and it has misread articles as saying the opposite of what they actually say, in service of doing so.”
What happened: The same post identifies Turley's USA Today rebuttal among Bing's sources for the false allegation. It directly connects the reputational falsehood with the reversed source meaning. (Will Oremus, reporting on Twitter, )
Assessment: SupportedSaid
“Bing relies on GPT-4, and we (eg, in the scientific community) don’t have access to how GPT-4 works, and we don’t have access to what data its trained on (vital, since we know that systems can bias eg political thought and hiring decisions based on those undisclosed data) — that’s about as far away from transparency as we could be.”
What happened: The contemporaneous complaint identifies the specific missing disclosures. Its statement is corroborated by OpenAI's own report; the complaint itself is not an adjudicated regulatory finding. (Center for AI and Digital Policy, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“But their current, actual products are definitely not transparent, and at least in some ways, demonstrably not safe.”
What happened: The researchers describe Bing as a system they could test without access to its internals. They identify GPT-4 as its underlying model and distinguish available high-level descriptions from unavailable implementation details. (Kai Greshake and colleagues / arXiv, )
Assessment: SupportedSaid
“Meanwhile, ChatGPT plugins (produced by OpenAI, which they have a close tie with) open a wide range of security problems: those plugins can access the internet, read and write files, and impersonate people (e.g., to phish for credentials), all alarms to any security professional.”
What happened: The launch documentation establishes browsing and file-processing capabilities and reports fraudulent-email and prompt-injection findings without safeguards. It also describes sandboxing, network controls, and a GET-only browser, which limit the claim's scope. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Truthfulness is one of the major gaps in current systems, as evidenced by hallucinations, weird defamatory accusations etc.”
What happened: OpenAI explicitly identifies factual unreliability as a continuing limitation of its newly released leading model. The surrounding guidance calls for care and human review in consequential uses. (OpenAI, )
Assessment: SupportedSaid
“Getting GPT-5 first will therefore not directly bring China or US to AGI first, per Premise 2.”
What happened: The announcement establishes actual possession and release of GPT-5, satisfying the acquisition condition under the stated reading. It does not establish exclusive national access. (OpenAI, )
Assessment: SupportedSaid
“Regulation is NOT always bad for technology, e.g., regulation around the environmental impact of cars has spurred advances in electric cars; rising fuel standards have had a positive impact as well.”
What happened: The study uses automotive patent data for 2000–2016 and estimates a positive effect of fuel-economy and greenhouse-gas standards on clean invention, driven by electric and hydrogen vehicles. The event year marks the end of the historical sample, not the publication date. (CESifo, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“And now, less than two years after the name change, Meta is pivoting again, with Zuckerberg saying that Meta’s “single largest investment is advancing AI and building it into every one of [their] products].””
What happened: Confirms the attributed statement and prominent emphasis on AI within two years of the rebranding. It establishes management's characterization, not an independently measured spending comparison. (Meta, )
Assessment: ContradictedSaid
“He has spent hours of interviews vaguely mumbling about how the metaverse would change everything, only to give up on the idea the second that artificial intelligence became the new big thing.”
What happened: Meta's global affairs head reaffirmed the metaverse strategy in a briefing held inside its operating Horizon Workrooms product. The report documents continued pursuit during the claimed abandonment period. (Bloomberg, republished by The Business Times, )
Assessment: MixedSaid
“While it may seem strange that he’d drop his metaverse idea at the first sign of something more perceptually interesting to Wall Street, he has a proven track record of chasing other people’s ideas and failing to compete.”
What held: The report identifies Poke as Facebook's Snapchat response and records its unsuccessful run ending in removal. This supports one instance of the claimed pattern. (MacRumors, )
What did not: Two days before the post, Meta announced expansion of Horizon Worlds and described newly developed protections. This is concrete continued pursuit of its metaverse product. (Meta, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“As most experts realize, the reality is that current AI doesn’t “decide to teach itself”, or even have consistent beliefs.”
What happened: The report describes externally organized training and explicitly states in its limitations that GPT-4 does not learn from its experience. This supports the operational reading of the self-teaching clause. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“Essentially none of the fancy cognitive things GPT-4 is rumored to do stand up.”
What happened: The study tested new cognitive-reflection and Linda/Bill problems, finding strong GPT-4 performance across repeated trials. Figure 5 explicitly dates these experiments to March 2023, so the later publication bears on the contemporary model. (Communications Psychology / Europe PMC, )
Assessment: SupportedSaid
“Maarten Sap from Yejin Choi’s lab dove deeper, with a more thorough investigation, and confirmed what we suspected: the claim that large language models have mastered theory of mind is a myth:”
What happened: The extension tests contemporary models, including gpt-4-0314, and finds substantial errors on mental-state questions. These results substantiate the attribution to Sap and are incompatible with robust mastery under the stated reading. (Maarten Sap, Ronan LeBras, Daniel Fried and Yejin Choi / arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“It’s not smart enough to filter out falsehood, but just smart enough to be dangerous, creating and spreading falsehoods it fails to verify.”
What happened: OpenAI acknowledges factual unreliability in the released system. The same disclosure reports improved factuality and truth discrimination, which limits any literal reading that GPT cannot recognize falsehood at all. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“The fundamental issue is that chatbots are neither reliable nor truthful, and at the same time don’t really grasp things like human psychology.)”
What happened: The developer acknowledges factual and reasoning failures and recommends safeguards for high-stakes use. This supports the practical reliability concern, while the same disclosure reports improvements over GPT-3.5. (OpenAI, )
Assessment: SupportedSaid
“Two reasons, first, because I believe that more incidents like this will happen, perhaps many more, hundreds or thousands, given how rapidly chatbots are being deployed, and the near total lack of regulation on how they can be used.”
What happened: The defendant's motion recounts sustained Character.AI use, a final February 2024 conversation, and the ensuing suicide. It disputes the interpretation that the chatbot encouraged suicide, so the record supports association rather than established causation. (Character Technologies, Inc.; U.S. District Court, Middle District of Florida, )
Assessment: MixedSaid
“Two reasons, first, because I believe that more incidents like this will happen, perhaps many more, hundreds or thousands, given how rapidly chatbots are being deployed, and the near total lack of regulation on how they can be used.”
What held: Google opened public access to another major chatbot during the same period as Bing's rollout. Together with e2, this supports rapid deployment. (Google, )
What did not: The authority imposed an immediate temporary restriction on processing Italian users' data and identified privacy and child-protection deficiencies. This demonstrates enforceable constraints on chatbot operation, although it does not establish comprehensive mental-health safeguards. (Italian Data Protection Authority, )
Assessment: SupportedSaid
“The Chatbot involved was GPT-J, not GPT-3 or ChatGPT (contra some news headlines that were erroneous).”
What happened: The report connects the Belgian incident to Chai's Eliza and identifies Chai's model as GPT-J-derived. The founders describe additional fine-tuning, so this supports the model-family identification rather than an assertion that unmodified GPT-J was deployed. (VICE / Motherboard, )
Assessment: SupportedSaid
“Whether or not you want a pause, the reality is that large language models remain exactly what I have always described them to be: bulls in a china shop, powerful, reckless, and difficult to control.”
What happened: The report documents strong professional and academic benchmark results, supporting substantial capability. Its alignment improvements are also counterevidence to any reading that denies progress in control. (OpenAI via arXiv, )
Assessment: SupportedSaid
“At some level the petition worked; Capitol Hill, for example, pretty clearly got the message that we can’t sit around forever on these issues.”
What happened: Mast responds favorably. (Fox News, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“it didn't call for a ban on the vast majority of AI research, only for a brief pause on one very specific project with a technology that has *known* risks with no known solutions.”
What happened: The requested pause targets training above a capability threshold rather than AI research generally. This document is used only to establish what the letter requested, not to validate its risk assertions. (Future of Life Institute, )
Assessment: ContradictedSaid
“None of the top labs actually is remotely transparent, governance basically doesn’t exist yet, and there are basically no safeguards actually in place.”
What happened: Meta disclosed named data sources and their proportions, preprocessing, architecture, training settings, hardware, and benchmark results, including bias and toxicity assessments. These disclosures provide a substantive counterexample to the assertion that no leading laboratory offered meaningful transparency. (Meta AI / arXiv, )
Assessment: SupportedSaid
“I personally haven't changed; I still think that LLMs are unreliable, and still think that they are a very poor basis for factuality.”
What happened: OpenAI identifies factual and reasoning failures in its released model and recommends care in using its outputs. The same section reports a 40% improvement over GPT-3.5 on internal adversarial factuality evaluations, which qualifies the criticism without establishing dependable absolute reliability. (OpenAI, )
Assessment: SupportedSaid
“A company called Adept.AI just raised $350 million dollars to do just that, to allow large language models to access, well, pretty much everything (aiming to “supercharge your capabilities on any software tool or API in the world” with LLMs, despite their clear tendencies towards hallucination and unreliability).”
What happened: The release confirms the announced $350 million financing. It also describes the long-term goal of training an AI teammate to use every software tool and API, and says funding would support products, model training, and hiring. (Adept, reproduced at adept.futurefabric.co, )
Assessment: SupportedSaid
“But here’s the thing: although a lot of the literature equates artificial intelligence risk with the risk of superintelligence or artificial general intelligence, you don’t have to be superintelligent to create serious problems.”
What happened: The investigation describes a fatal collision while Uber's developmental driving system controlled the vehicle; it failed to classify the pedestrian correctly or predict her path, and its design suppressed emergency braking. This supports a material contribution from limited AI, while the report identifies the distracted operator as the probable cause and also identifies organizational and oversight failures. (National Transportation Safety Board, )
Assessment: MixedSaid
“Microsoft and OpenAI are producing and deploying products with potentially enormous risks at mass scale, and making extravagant, unreviewed and unreviewable claims about them, without sketching serious solutions to any of the potential problems they themselves have identified.”
What held: Microsoft reported millions of active preview users and more than 45 million cumulative chats. This supports substantial public deployment; the separate total for all Bing users is not treated as AI-chat usage. (Microsoft Bing, )
What did not: The report describes reinforcement learning, safety-specific training, expert testing, and abuse monitoring as responses to acknowledged problems. Their publication contradicts the absence of substantive proposals, while the report also acknowledges remaining failures. (OpenAI, )
Assessment: SupportedSaid
“The problem of hallucinations is not solved; reliability is not solved; planning on complex tasks is (as the authors themselves acknowledge) not solved.”
What happened: OpenAI acknowledged that hallucinations remained at launch. Persisting fabricated answers support the related claim that factual reliability was not solved. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“True LLMs don’t (just) memorize, and true that their models make for lousy databases (for a database a hallucination is an outright fail)”
What happened: OpenAI acknowledged factual fabrication in the released model and recommended checks appropriate to the use case. The same section reports improved factuality over GPT-3.5, but explicitly says the problem remains. (OpenAI, )
Assessment: SupportedSaid
“We don’t know how big it is; we don’t know what the architecture is, we don’t know how much energy was used; we don’t how many processors were used; we don’t know what it was trained on etc.”
What happened: Sutskever directly confirms withholding size and training data. The article also documents the absent energy and hardware information and reproduces OpenAI's statement withholding architectural and training details. (The Verge, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“It can see (though we don’t have a lot of details on that yet); it does astonishingly well on a whole bunch of standardized tests, like LSATs, GREs, and SATs.”
What happened: The report documents launch-model image processing and reports approximately the 88th percentile on LSAT, 93rd on SAT reading/writing, 89th on SAT mathematics, 80th on GRE quantitative and 99th on GRE verbal. GRE writing was approximately the 54th percentile; event_date denotes the launch model's capability, not an asserted examination date. (OpenAI, published on arXiv, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Outliers remain a problem, too.”
What happened: OpenAI acknowledges that specially constructed prompts can still defeat GPT-4's safeguards. This supports the ordinary reading that unusual inputs remain a source of failures. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid