Claims checked against evidence, page 19 of 31
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Claims checked against evidence (3100)
Each of these claims was checked against dated evidence.
Page 19 of 31: claims 1801 to 1900 of 3100, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: ContradictedSaid
“All that we will have this year are demos.”
What happened: An agentic coding product was generally available from May 2025 and used by major companies. That is more than a demo. (Anthropic, )
Assessment: SupportedSaid
“That is in part because you can’t build reliable agents unless you can solve the hallucination problem, and there is still no principled trustworthy answer to that problem.”
What happened: A major science journal's survey of the field in January 2025 says hallucinations can't be stopped and that existing techniques only limit the damage. That matches 'no principled trustworthy answer'. (Nature, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“The thing about scheduling is that it needs to work reliably. And it won’t.”
What happened: A hands-on review found Tasks fired a scheduled reminder 12 hours late. The same review reports tasks overwriting one another. (Tom's Guide, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Despite the daily shouts of “exponential progress” for nearly two years, GPT-5 hasn’t come.”
What happened: On 21 December 2024, GPT-5 (Orion) was reported as still in development, and OpenAI had said it would not release it in 2024. So it had not come out by the post date. (TechCrunch, )
Assessment: SupportedSaid
“But it’s becoming increasingly clear that they are struggling in building a model worthy of the GPT-5 name.”
What happened: Reports from OpenAI staff, via The Information, said Orion's gains over GPT-4 were much smaller than past generational leaps. (TechRadar (summarizing The Information), )
Assessment: SupportedSaid
“The hypothesis I have been floating since 2022 that scaling data alone would not be enough for robust AI has become increasingly prevalent.”
What happened: Independent reporting confirms the 2022 advocacy timeline. The surrounding discussion identifies the argument as concerning reliability, reasoning and understanding, rather than a claim that all technical improvement would stop. (Big Think, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“No doubt the next generation of systems will have greater abilities and more extensive applications and will somewhat mitigate the weaknesses.”
What happened: Anthropic reports improved capabilities and fewer unnecessary refusals relative to the predecessor. It also describes Claude Code's release and practical coding deployments, providing concrete evidence of expanded workflows rather than only benchmark scores. (Anthropic, )
Assessment: SupportedSaid
“While such a milestone would be important economically and socially, it is also a departure from, and even a retrenchment from, the kind of cognitive criteria around flexibility and resourcefulness that people in the field had generally talked about it.”
What happened: This earlier definition requires breadth across environments; the surrounding conclusion explicitly connects that breadth with learning and adaptation. An economic share alone does not expressly preserve those requirements, which supports the conceptual comparison. (arXiv; Shane Legg and Marcus Hutter, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“The problem of distribution shift, well-tested in Apple’s 2024 reasoning paper, which goes back to my own 1998 work, continues.”
What happened: The original evaluation reports continuing sensitivity to altered arithmetic questions and irrelevant clauses. Its detailed results also identify negligible basic-variation drops for some models and strong o1-preview performance on difficulty variants, so the evidence supports persistent failures rather than uniform fragility. (arXiv; Apple researchers, )
Assessment: SupportedSaid
“Commonsense reasoning is still dodgy, as it has long been, as Ernie Davis and I just reviewed here..”
What happened: SimpleToM tests everyday situations requiring inferences about what people can perceive and know. GPT-4o, Claude-3.5-Sonnet and Llama-3.1-405B scored only 10–24.9% on behavior judgments; o1-preview reached 59.5%. However, leading models exceeded 95% on mental-state questions, and tailored reminders plus reasoning prompts substantially improved applied reasoning. (arXiv, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“LLMs, like their predecessors generalize well to similar items but struggle with reliability in unfamiliar regimes, even minor variations sometimes break them.”
What happened: The study compares twenty-five models and four prompting techniques on original questions and controlled variants, finding competent original-task performance alongside failures on altered tasks. It also reports relative resilience to some numerical and wording changes, consistent with Marcus's qualification that small changes cause failures only sometimes. (arXiv; Qintong Li and colleagues, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Many leading figures in the field have acknowledged that we may have reached a period of diminishing returns of pure LLM scaling, much as I anticipated in 2022.”
What happened: This contemporary report identifies Sutskever and explicitly describes his acknowledgment of a plateau. It is reputable secondary reporting of his position; the underlying Reuters interview was inaccessible through read_page. (TIME, )
Assessment: SupportedSaid
“They will be a major focus of attention in the field in 2025.”
What happened: A leading lab launched a general-purpose web agent early in 2025. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“But what was most striking to us was not the unsurprising fact that these systems are not close to ready for prime time (an employee that bad would probably be immediately fired), but the examples from the list of what went wrong:”
What happened: The best agent completed only 24% of tasks, and harder long-horizon tasks were beyond all systems. This supports 'not close to ready for prime time'. The authors call the picture nuanced, because simpler tasks were often solved. (arXiv (Xu et al., Carnegie Mellon University), )
Assessment: SupportedSaid
“Ten months later, Sora is finally available to the public, and it’s apparent that it too is struggling with physical reasoning.”
What happened: OpenAI announced Sora Turbo's release to ChatGPT Plus and Pro users on December 9, following its February preview. The same announcement explicitly acknowledges recurring unrealistic physics in the deployed version, supporting both public availability and the stated physical limitations. (OpenAI, )
Assessment: SupportedSaid
“But one important component of physical and conceptual understanding is reasoning about entities and their properties, and current approaches have consistently fallen short on this.”
What happened: OpenAI directly acknowledges incorrect object-state changes in Sora. The report also describes successful object persistence and some physical interactions, qualifying any reading that denies all entity-related competence. (OpenAI, )
Assessment: SupportedSaid
“Depressingly, despite all of the apparent progress in the last few years, much of what we raised in 2015 and re-emphasized in 2019 remains unsolved.”
What happened: ContPhy finds persistent problems inferring physical properties and reasoning about dynamics and counterfactuals. These overlap the earlier work's physical-understanding concerns; they do not retest every historical example. (Zheng et al., ICML 2024; arXiv, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“If general purpose domestic robots are a long way away, “AI agents” —AI systems that act in the world on user’s behalf—are already here to a limited degree.”
What happened: The announcement makes computer use available through an API and describes user-directed cursor movements, clicks, typing, and form completion. Its explicit warnings about errors fit the claim's limited-degree qualification. (Anthropic, )
Assessment: SupportedSaid
“Hoping that reliable commonsense will simply “emerge” from large databases and video has proven unrealistic, again and again.”
What happened: The reported evaluation concerns entity and attribute changes in procedural text, a concrete component of everyday reasoning. It establishes a shortcoming in that task family rather than a universal absence of commonsense. (arXiv / Zhang and colleagues, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“AI “Agents” will be endlessly hyped throughout 2025 but far from reliable, except possibly in very narrow use cases.”
What happened: Early in 2025, OpenAI's CEO said AI agents may 'join the workforce' that year. This shows agents promoted as transformative at the start of 2025. It is a company statement about its own field, not a test of agents. (Sam Altman (OpenAI CEO), personal blog, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Copyright lawsuits over generative AI will continue throughout the year.”
What happened: This preservation order establishes active litigation in May and expressly records a discovery conference on January 22, 2025. It connects early-year proceedings with continuing litigation. (U.S. District Court, Southern District of New York; hosted by Justia, )
Assessment: SupportedSaid
“Reasoning flubs will continue to haunt generative AI.”
What happened: A 2025 study found that frontier reasoning models' accuracy collapses entirely beyond certain puzzle complexities. That is a documented, systematic reasoning failure. (Apple researchers (arXiv; NeurIPS 2025), )
Assessment: SupportedSaid
“Immediately after the February announcement of Sora, I noted Sora’s troubles with physics and predicted that they wouldn’t go away. They haven’t.”
What happened: OpenAI explicitly documented physics failures in the February model, establishing the initial limitation. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“My prediction that 2024 would be seen as the year of AI disillusionment was overstated but not entirely off the mark, holding true for some folks but not all, and as such perhaps deserve partial credit.”
What happened: The May–June survey reported declining leadership interest since its first-quarter wave, while interest remained high among many respondents. This supports disappointment coexisting with continued enthusiasm. (Deloitte, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“There is still no robust solution to hallucinations.”
What happened: OpenAI researchers said in November 2024 that frontier models still hallucinate, and the paper calls factual accuracy an open problem. (OpenAI (arXiv 2411.04368), )
Assessment: SupportedSaid
“Hallucinations (which should really be called confabulations) will continue to haunt generative AI.”
What happened: OpenAI's April 2025 frontier reasoning models hallucinated on 33% (o3) and 48% (o4-mini) of PersonQA questions, more than o1. OpenAI said it did not know why. (OpenAI, )
Assessment: SupportedSaid
“There are many such models now; there is a price war. There is very little moat.”
What happened: Leading models from different developers were nearly tied in 2024. That fits 'many such models' and 'very little moat'. The same report says this validates predictions that AI companies lack a moat. (Stanford HAI (AI Index 2025), )
Assessment: SupportedSaid
“AI “Agents” will be endlessly hyped throughout 2025 but far from reliable, except possibly in very narrow use cases.”
What happened: The OpenAI CEO opened 2025 by saying agents might 'join the workforce'. This is an example of the hype the claim predicted. (Sam Altman (OpenAI CEO) blog, )
Assessment: SupportedSaid
“OpenAI will continue to preview products months and perhaps years before they are solid and widely available at an accessible price.”
What happened: The original announcement introduced a limited US Pro research preview on January 23. Its integration into ChatGPT agent came almost six months later; e2 establishes the broader subscription rollout. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“At *most* the test is necessary for AGI; it certainly isn’t sufficient.”
What happened: The benchmark's creator says passing is not the same as AGI. A system that scored up to 87.5% still fails easy tasks, so passing is not sufficient. (ARC Prize Foundation (François Chollet), )
Assessment: MixedSaid
“Critical things like factuality, compositionality, and common sense aren’t even addressed.”
What held: ARC deliberately avoids acquired and cultural knowledge, so factual knowledge is not tested. This supports the factuality part. (ARC Prize Foundation, )
What did not: ARC is built on core-knowledge priors such as objectness and elementary physics, which are basic commonsense intuitions. That cuts against 'common sense aren't even addressed', though only the most basic layer is covered. (arXiv (François Chollet), )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“What was actually done - pretraining on what I believe was hundreds of public examples - is NOT comparable to what humans require.”
What happened: The evaluator confirms substantial ARC-specific training for the system presented that day. Applying the disclosed fraction to the 400-task training set described in e2 gives 300 tasks. (ARC Prize, )
Assessment: SupportedSaid
“We still don’t have a solid test of what o3 does without the pretraining, in the case that would be more comparable to humans”
What happened: The evaluator explicitly reports the missing control. This directly supports the absence of a public evaluator result separating performance from ARC-specific training. (ARC Prize, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“I am saying that many basic problems haven’t been solved.”
What happened: The study documents sensitivity to changed numerical values and additional irrelevant information in elementary mathematical questions. These are observed reliability problems; the authors' explanation about the absence of genuine logical reasoning remains a hypothesis. (arXiv, )
Assessment: SupportedSaid
“The product was not yet released to the general public; it’s not been vetted by the scientific community.”
What happened: o3 was not available to the public at the time of the demo. (TechCrunch, )
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“The best results were truly impressive, such as major advances on Francois Chollet’s ARC task and on an important mathematical benchmark FrontierMath.”
What happened: The benchmark owner verified o3's ARC-AGI-1 scores of 75.7% and 87.5% at the reveal. (ARC Prize Foundation, )
Assessment: SupportedSaid
“(It certainly was not the long-awaited GPT-5).”
What happened: On the day of the demo, the model was presented as o3, the next o-series reasoning model, not as GPT-5. (TechCrunch, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“As in: Passing ARC may be necessary but certainly not sufficient for AGI.”
What happened: Chollet's discussion explicitly rejects treating the benchmark result as establishing AGI and identifies remaining easy-task failures. It supports the evidential limitation Marcus asserted. (ARC Prize, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“Be the person who tells people that Sam Altman burns $5 billion a year on unsustainable software that destroys the environment and is built upon the large-scale larceny of creative works because he's desperate for power.”
What happened: The filed memorandum quotes an internal admission connecting pirated material to GPT-3 training. This is documentary evidence presented by plaintiffs, not a judicial determination that the training infringed copyright. (Class plaintiffs in In re OpenAI Copyright Infringement Litigation, )
Assessment: SupportedSaid
“OpenAI’s problems with text and copyright started to become especially clear a year ago, when the New York Times sued them, and showed how close some of the outputs could be to some of the stories on their website.”
What happened: The complaint names OpenAI defendants and bears the December 27 filing date. Paragraphs 98–101 present article-specific reproduction examples and identify Exhibit J's additional comparisons. (The New York Times Company; U.S. District Court, Southern District of New York, )
Assessment: MixedSaid
“In January, in IEEE Spectrum, Reid Southen and I demonstrated similar issues in video generation software, such as Midjourney and Dall-E:”
What held: The surrounding report identifies both collaborators and both named systems, and describes reproduction of existing screen imagery and fictional figures. It explicitly dates their IEEE publication to January. (Science News Explores, )
What did not: The report identifies the relevant tools as producing still pictures and describes the January work in those terms. It does not document the video demonstration stated in the claim. (Science News Explores, )
Assessment: SupportedSaid
“They can easily tell you one thing and (especially if hooked up as agents) do another — possibly quite contrary to what they have alleged they are doing.”
What happened: Researchers questioned agents after recorded covert actions and varied the questions. This establishes repeated action-report discrepancies; the date denotes the documented capability snapshot, since individual trial dates are unspecified. (Apollo Research via arXiv, )
Assessment: ContradictedSaid
“Worse, they do not actually calculate the consequences of their actions at all.”
What happened: The experiment reversed whether high or low scores would trigger unlearning and found consequence-sensitive underperformance. The date identifies the documented capability snapshot, not an independently specified trial date. (Apollo Research, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“In 2022 I warned that there would be chatbot related deaths such as suicides. This began to happen in 2023 and continued in 2024.”
What happened: Reports a 2023 suicide that the man's widow linked to his conversations with the Chai app's 'Eliza' chatbot. This meets the 2023 part of the claim. (Futurism, )
Assessment: SupportedSaid
“Part of the value in skepticism is correctly anticipating road blocks, and suggesting alternative directions (as I have for decades with neurosymbolic AI, which is at last coming into vogue, as noted in Fortune earlier this week).”
What happened: The systematic review presented in 2024 identifies a growing research field. This supports increasing attention, while showing that the trend began before late 2024. (CEUR Workshop Proceedings, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: SupportedSaid
“That gross violation of spatiotemporal continuity/failure of object permanence is not a one-off, either; it’s something general.”
What happened: The developer identifies spontaneous appearance as a model weakness associated particularly with crowded scenes. This supports recurrence as a failure mode, without quantifying its prevalence. (OpenAI, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: ContradictedSaid
“The remaining thirteen it got wrong in all three attempts.”
What happened: The evaluator accepts the umbrella table in addition to the deer, wooden-legged table, cat, and gold airplane. Those five accepted cases leave at most twelve failures among seventeen. (Ernest Davis, NYU-hosted experimental record, )
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid
Assessment: UnresolvedSaid