Claims checked against evidence, page 16 of 31
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Claims checked against evidence (3100)
Each of these claims was checked against dated evidence.
Page 16 of 31: claims 1501 to 1600 of 3100, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
“In 2024 alone, 31 states, Puerto Rico, and the Virgin Islands enacted AI-related legislation or resolutions, and at least 27 states passed deepfake laws.”
What happened: The legislators' letter states both numbers and immediately identifies examples as 2024 enactments. It predates Marcus's post and is not his own writing. (IAPP; authored by the Multistate AI Policymaker Working Group, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“But the problem is not going away; OpenAI’s recent o3 actually hallucinates more than some its predecesssors.”
What happened: The release report gives PersonQA hallucination rates of 0.33 for o3 and 0.16 for o1, and SimpleQA rates of 0.51 and 0.44. It also reports higher accuracy for o3 on both tests, so increased hallucination does not mean uniformly worse performance. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“But they don’t operate like people. They don’t, for example, ever fact check (as humans sometimes, when well motivated, do). They mimic the kinds of things of people say in various contexts. And that’s essentially all they do.”
What happened: RARR uses PaLM to compare generated assertions with retrieved evidence and edits the passage when disagreement is detected. Its evaluated research-and-revision procedure is a counterexample to the categorical claim under the functional reading. (Association for Computational Linguistics; author manuscript hosted by arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“And they have such a superficial understanding of their own output that they can’t begin to fact check it.”
What happened: In the no-retrieval self-evaluation setting, Table 5 reports correct assessments for 201 of 532 false GPT-generated claims and 229 of 537 false GLM-generated claims. Substantial abstention remains, but identifying these errors contradicts the claimed absence of meaningful self-fact-checking ability. (Jiayi Yao, Haibo Sun and Nianwen Xue / arXiv, )
Assessment: UnresolvedSaid
Assessment: MixedSaid
“METR’s blog and its tweets (perhaps written by publicists or by generative AI rather than by scientists), however, dropped the careful qualifications and nuance of the technical paper and made claims that go far beyond anything that the study actually supports.”
What held: The accompanying text announces a roughly seven-month doubling of task length without stating the software-task restriction or 50% success threshold. This supports the narrower complaint about qualifications omitted from the opening tweet, rather than establishing that an entire thread lacked them. (METR, )
What did not: The dated reproduction preserves this warning, methodological dependence, conditional extrapolation, and discussion of external-validity uncertainty. It contradicts a general claim that the blog discarded the paper’s qualifications. (LessWrong, reproduction by Zach Stein-Perlman, )
Assessment: SupportedSaid
“Computers can answer some questions reliably now, for example, and some not, and the graph tells us nothing about which is which or when any specific question will be solved.”
What happened: The defined output is a duration associated with aggregate success probability. The paper’s fitting and extrapolation methods do not turn this into a list of reliably answered questions or dates for solving identified questions. (METR researchers via arXiv, )
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“It also, at this point, is pretty obvious that generative AI isn't going to do much more than it does today.”
What happened: The independently verified result demonstrates a large increase in successful abstract-reasoning task completion over the April baseline. The report also cautions that refinement harnesses are domain-specific and that benchmark-domain training may contribute to performance. (ARC Prize Foundation, )
Assessment: ContradictedSaid
“The expectation that generative AI will transform into something much, much more powerful requires you to first ignore the existing limitations, believing it to be more capable than it is, and also ignore the fact that these models have yet to show meaningful improvement over the past few years.”
What happened: The original evaluation finds substantial growth in the duration of software and reasoning tasks agents can complete with 50% success. Its historical comparisons include the years preceding the post, although this is a restricted task distribution. (METR researchers, published on arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Anthropic is a business (which incidentally neglects to respect the rights of artist and writers who work they nick).”
What happened: The order records Anthropic's knowing acquisition of pirated books in July 2022 and rejects fair use for the central-library copies. The event date refers to the historical acquisition; the legal assessment was published later. (United States District Court for the Northern District of California, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“the crown jewel that they reported on the demo — the 75% on Francois Chollet’s ARC test (once called ARC-AGI) doesn’t readily replicate.”
What happened: As reported April 22, o3-medium scored 53% on ARC-AGI-1's semi-private set, while o3-high returned only 37 of 100 tasks; the evaluator rejected its 82% returned-task accuracy as unrepresentative. The same report affirms the December result but explains that production o3 differed in model, training and available compute, supporting practical non-reproduction rather than invalidity of the original experiment. (ARC Prize, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“OpenAI had privileged access to data their competitors didn’t have, but didn’t acknowledge this.”
What happened: Epoch retrospectively states that OpenAI owned the commissioned problems and could access problems and solutions subject to a holdout exception, while sharing with other parties required OpenAI's written permission. It distinguishes the partnership disclosure around the o3 announcement from the data-access arrangement, which it acknowledges was not clarified then. (Epoch AI, )
Assessment: SupportedSaid
“without even acknowledging, for example, the massive problem LLMs have with fabricating citations”
What happened: The researchers documented invented source links across multiple systems, supporting a substantial citation-reliability problem. Their reported majority rate combines fabricated and broken URLs, so it is not a measured fabrication-only rate. (Tow Center for Digital Journalism / Columbia Journalism Review, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“It certainly took zero responsibility for what appears to be an obvious infringement.”
What happened: The quoted opening is followed by general arguments about creativity, copyright boundaries, and possible compensation. Across the complete section, Altman neither accepts fault for the Peanuts image nor offers a specific remedy. (The Singju Post, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“OpenAI, as I've written before, is effectively the entire generative AI industry, with its nearest competitor being less than five percent of its 500 million weekly active users.”
What happened: The decisive quantity is a derived weekly-user lower bound: the same day’s active users must also be active in its containing week. The reported 35 million therefore exceeds the claimed 25-million ceiling without equating daily and weekly counts. (NPR via Georgia Public Broadcasting, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“SoftBank's exposure to OpenAI is materially harming the company.”
What happened: JCR changed the outlook after discussing the large OpenAI commitment, Stargate and increased exposure to unlisted AI assets. This supports deterioration in the credit profile, while the same sentence confirms that the rating itself was affirmed. (Japan Credit Rating Agency, )
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“OpenAI has become a load-bearing company for the tech industry, both as a narrative — as previously discussed, ChatGPT is the only Large Language Model company with any meaningful userbase — and as a financial entity.”
What happened: Meta reported substantial recurring use of its AI assistant during the target period. This contradicts the ordinary reading that only ChatGPT had any meaningful userbase, without requiring a direct comparison to ChatGPT's weekly usage. (Meta Platforms, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“GenAI is pitched as (near) AGI, but reliability problems persist, limiting how well it can be used in practice.”
What happened: This is primary evidence of a prominent supplier promoting progress toward AGI. It supports the promotional premise, not the proposition that any existing product actually was AGI. (Sam Altman, OpenAI CEO, )
Assessment: SupportedSaid
“My sources assure me that this is NOT a clerical error and that Papan’s office confirms that the change is real.”
What happened: The official journal records adoption of the author's amendments to AB 501. This establishes an actual legislative action rather than merely an altered webpage. (California State Assembly, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“First, the AI community is in fact pretty disappointed with LLama 4’s performance:”
What happened: The report records Willison's assessment alongside complaints about practical performance from researchers and developer communities. It supports visible disappointment, without measuring its prevalence. (Ars Technica, )
Assessment: ContradictedSaid
“If the juicing of benchmarks and “blending” of test data is the primary scandal, there is a secondary scandal too: a bunch of well-paid pundits, such as Roose, Newton, and Cowen, are failing to reckon with the repeated failures of massive companies to build GPT-5.”
What happened: Newton explicitly covered setbacks at OpenAI, Google and Anthropic, then examined data limits, benchmark gaming, the possible insufficiency of LLMs, and alternative routes to progress. This is substantive engagement even though he remained optimistic. (Platformer — Casey Newton, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“NVidia is down by a third, so far in 2025. (Far more than the stock market itself.)”
What happened: The reported decline is close to one third, though it is not literally 33.33%. Financial reporting is used because the historical-price filing could not be read. (TheStreet, syndicated by Yahoo Finance, )
Assessment: SupportedSaid
“there still ain’t no GPT-5”
What happened: The full announcement schedules GPT-5 several months ahead and describes unfinished integration. This is direct evidence that GPT-5 remained forthcoming immediately before Marcus's post. (Sam Altman, OpenAI, )
Assessment: SupportedSaid
“models still don’t reason reliably (as anticipated in What to Expect When Your Expecting GPT-4)”
What happened: Experts graded complete solutions to six newly released USAMO problems from six reasoning models, including o1-pro, o3-mini, Claude 3.7, and DeepSeek R1. The reported percentage concerns graded performance, not a general accuracy rate for all reasoning tasks. (arXiv; Petrov and colleagues, )
Assessment: ContradictedSaid
“Scaling has run out (like I said it would)”
What happened: OpenAI ties the larger pretraining run to gains: its table reports GPQA scores of 71.4% versus 53.6% for GPT-4o and AIME 2024 scores of 36.7% versus 9.3%. These are company-reported comparisons; the release also acknowledges architectural and optimization changes. (OpenAI, )
Assessment: SupportedSaid
“A new study on math, supporting what Davis and I wrote yesterday re LLMs struggling with mathematical reasoning from Mahdavi et al, converges on similar conclusions”
What happened: The original abstract contains the conclusions Marcus attributes to the study. They align directly with concerns about invalid proofs and unreliable recognition of mathematical errors. (Mahdavi et al., arXiv, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“The important problem is that they sometimes claim to they have solved problems that they have not properly solved; as with hallucinations, this reflects a deeply problematic failure to sanity check their own work.”
What happened: The study pairs claims of completion with expert findings of flawed logic, unjustified steps and other proof defects. Its independent double-grading procedure supports the limited claim that models sometimes deliver incorrect work as solved. (arXiv, )
Assessment: SupportedSaid
“The results were dismal: None of the AIs scored higher than 5% overall.”
What happened: Table 1 reports average totals of 2.0, 1.8, 1.5, 1.2, 1.2 and 0.9 points for the six named models, against a maximum of 42. The highest reported total therefore corresponds to approximately 4.76%, below 5%. (Petrov et al., ETH Zurich and INSAIT; arXiv, )
Assessment: SupportedSaid
“But that is because they rely in part on powerful, completely hand-written, symbolic reasoning systems. LLMs are not similarly immune.”
What happened: The paper describes AlphaProof's use of Lean and attributes proof correctness to kernel verification. It also explains that informal LLM reasoning lacks guaranteed correctness; this supports the mechanism behind Marcus's comparison. (Nature, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“In Taming Silicon Valley, I warned that LLMs would be used for dumb things that would affect lots of people.”
What happened: This is primary textual evidence reproduced within a secondary publication. It establishes that the book raised concern about consequential, potentially biased uses of LLMs; it does not establish that those uses actually occurred. (Big Think, republishing an excerpt from The MIT Press book, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“To do well the system had to be coached to adopt the persona of “a young person who is introverted, knowledgeable about internet culture, and uses slang”; it’s still in no small part about the tricks.”
What happened: The randomized comparison reports GPT-4.5 identification rates of 73% with the persona and 36% without it, with persona performance above chance in both populations. However, no-persona performance was not significantly below chance in the Prolific population, which limits any absolute claim of necessity. (arXiv; Cameron R. Jones and Benjamin K. Bergen, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“But in an initial public offering yesterday CoreWeave valued themselves at just $23 billion - a major retreat.”
What happened: Reuters directly reports both valuations on the fully diluted basis. The approximate reduction is 28.1%, which reasonably supports the description of a major retreat. (Reuters, syndicated by Yahoo Finance, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Especially since DeepSeek there has been a massive price war.”
What happened: Alibaba announced a large reduction within the target period. The maximum discount must not be read as a reduction across every model. (Reuters, syndicated by WHBL, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“GenAI is an approach that is enormously popular, but one that is neither reliable nor particularly well-grounded in truth.”
What happened: Company-reported adoption supports popularity. (TechCrunch, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“That’s no discussion of the massive, world-changing consequences of a (again, totally theoretical, no proof this exists) artificial intelligence that's as smart and capable (again, how is it capable?) as a human being.”
What happened: The scientific synthesis describes uneven capabilities and continuing reasoning limitations rather than established general human equivalence. This supports the contextual reading when combined with the practical evaluation in e2. (International AI Safety Report, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“I can find no way to explain this other than that Microsoft has decided it doesn't want to be OpenAI's compute provider any further, and that any further expansion that OpenAI needs will be its responsibility to acquire.”
What happened: The same paragraph expressly states that OpenAI will increase Azure consumption while adding Stargate compute. This provides a contemporaneous alternative to the claim that Microsoft had abandoned further supply growth. (OpenAI and SoftBank, )
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“I estimate OpenAI will lose as much as $14 billion in 2025.”
What happened: The reported annual net loss exceeds the $14 billion upper-end estimate. The article attributes much of the gap to restructuring accounting and reports a substantially smaller adjusted figure; it describes Financial Times review of the records, rather than relying solely on Zitron's conclusion. (Ars Technica, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Many other authors are discovering the same thing; social media is full of reports like mine.”
What happened: Katsu explicitly places the start of repeated author discovery posts on Thursday, March 20. Her account also covers Friday; only its Thursday component bears on the target period. (Alma Katsu, )
Assessment: SupportedSaid
“He worked hard on every one of those books and articles and does not appreciate them being used without permission by AI systems with an established tendency towards regurgitation.”
What happened: The experiments include LLaMA 7B and 65B and find repeated matching passages. For models with undisclosed training corpora, the researchers use an auxiliary corpus, which limits certainty about individual passages' training provenance. (Nasr and colleagues, arXiv, )
Assessment: UnresolvedSaid