Supported claims, page 8 of 9
Claims by Gary Marcus and Ed Zitron. Deadlines are judged as of 2026-10-29.
Supported claims (832)
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Page 8 of 9: claims 701 to 800 of 832, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
“Getting GPT-5 first will therefore not directly bring China or US to AGI first, per Premise 2.”
What happened: The announcement establishes actual possession and release of GPT-5, satisfying the acquisition condition under the stated reading. It does not establish exclusive national access. (OpenAI, )
Assessment: SupportedSaid
“Regulation is NOT always bad for technology, e.g., regulation around the environmental impact of cars has spurred advances in electric cars; rising fuel standards have had a positive impact as well.”
What happened: The study uses automotive patent data for 2000–2016 and estimates a positive effect of fuel-economy and greenhouse-gas standards on clean invention, driven by electric and hydrogen vehicles. The event year marks the end of the historical sample, not the publication date. (CESifo, )
Assessment: SupportedSaid
“And now, less than two years after the name change, Meta is pivoting again, with Zuckerberg saying that Meta’s “single largest investment is advancing AI and building it into every one of [their] products].””
What happened: Confirms the attributed statement and prominent emphasis on AI within two years of the rebranding. It establishes management's characterization, not an independently measured spending comparison. (Meta, )
Assessment: SupportedSaid
“As most experts realize, the reality is that current AI doesn’t “decide to teach itself”, or even have consistent beliefs.”
What happened: The report describes externally organized training and explicitly states in its limitations that GPT-4 does not learn from its experience. This supports the operational reading of the self-teaching clause. (OpenAI, )
Assessment: SupportedSaid
“Maarten Sap from Yejin Choi’s lab dove deeper, with a more thorough investigation, and confirmed what we suspected: the claim that large language models have mastered theory of mind is a myth:”
What happened: The extension tests contemporary models, including gpt-4-0314, and finds substantial errors on mental-state questions. These results substantiate the attribution to Sap and are incompatible with robust mastery under the stated reading. (Maarten Sap, Ronan LeBras, Daniel Fried and Yejin Choi / arXiv, )
Assessment: SupportedSaid
“It’s not smart enough to filter out falsehood, but just smart enough to be dangerous, creating and spreading falsehoods it fails to verify.”
What happened: OpenAI acknowledges factual unreliability in the released system. The same disclosure reports improved factuality and truth discrimination, which limits any literal reading that GPT cannot recognize falsehood at all. (OpenAI, )
Assessment: SupportedSaid
“The fundamental issue is that chatbots are neither reliable nor truthful, and at the same time don’t really grasp things like human psychology.)”
What happened: The developer acknowledges factual and reasoning failures and recommends safeguards for high-stakes use. This supports the practical reliability concern, while the same disclosure reports improvements over GPT-3.5. (OpenAI, )
Assessment: SupportedSaid
“Two reasons, first, because I believe that more incidents like this will happen, perhaps many more, hundreds or thousands, given how rapidly chatbots are being deployed, and the near total lack of regulation on how they can be used.”
What happened: The defendant's motion recounts sustained Character.AI use, a final February 2024 conversation, and the ensuing suicide. It disputes the interpretation that the chatbot encouraged suicide, so the record supports association rather than established causation. (Character Technologies, Inc.; U.S. District Court, Middle District of Florida, )
Assessment: SupportedSaid
“The Chatbot involved was GPT-J, not GPT-3 or ChatGPT (contra some news headlines that were erroneous).”
What happened: The report connects the Belgian incident to Chai's Eliza and identifies Chai's model as GPT-J-derived. The founders describe additional fine-tuning, so this supports the model-family identification rather than an assertion that unmodified GPT-J was deployed. (VICE / Motherboard, )
Assessment: SupportedSaid
“Whether or not you want a pause, the reality is that large language models remain exactly what I have always described them to be: bulls in a china shop, powerful, reckless, and difficult to control.”
What happened: The report documents strong professional and academic benchmark results, supporting substantial capability. Its alignment improvements are also counterevidence to any reading that denies progress in control. (OpenAI via arXiv, )
Assessment: SupportedSaid
“At some level the petition worked; Capitol Hill, for example, pretty clearly got the message that we can’t sit around forever on these issues.”
What happened: Mast responds favorably. (Fox News, )
Assessment: SupportedSaid
“it didn't call for a ban on the vast majority of AI research, only for a brief pause on one very specific project with a technology that has *known* risks with no known solutions.”
What happened: The requested pause targets training above a capability threshold rather than AI research generally. This document is used only to establish what the letter requested, not to validate its risk assertions. (Future of Life Institute, )
Assessment: SupportedSaid
“I personally haven't changed; I still think that LLMs are unreliable, and still think that they are a very poor basis for factuality.”
What happened: OpenAI identifies factual and reasoning failures in its released model and recommends care in using its outputs. The same section reports a 40% improvement over GPT-3.5 on internal adversarial factuality evaluations, which qualifies the criticism without establishing dependable absolute reliability. (OpenAI, )
Assessment: SupportedSaid
“A company called Adept.AI just raised $350 million dollars to do just that, to allow large language models to access, well, pretty much everything (aiming to “supercharge your capabilities on any software tool or API in the world” with LLMs, despite their clear tendencies towards hallucination and unreliability).”
What happened: The release confirms the announced $350 million financing. It also describes the long-term goal of training an AI teammate to use every software tool and API, and says funding would support products, model training, and hiring. (Adept, reproduced at adept.futurefabric.co, )
Assessment: SupportedSaid
“But here’s the thing: although a lot of the literature equates artificial intelligence risk with the risk of superintelligence or artificial general intelligence, you don’t have to be superintelligent to create serious problems.”
What happened: The investigation describes a fatal collision while Uber's developmental driving system controlled the vehicle; it failed to classify the pedestrian correctly or predict her path, and its design suppressed emergency braking. This supports a material contribution from limited AI, while the report identifies the distracted operator as the probable cause and also identifies organizational and oversight failures. (National Transportation Safety Board, )
Assessment: SupportedSaid
“The problem of hallucinations is not solved; reliability is not solved; planning on complex tasks is (as the authors themselves acknowledge) not solved.”
What happened: OpenAI acknowledged that hallucinations remained at launch. Persisting fabricated answers support the related claim that factual reliability was not solved. (OpenAI, )
Assessment: SupportedSaid
“True LLMs don’t (just) memorize, and true that their models make for lousy databases (for a database a hallucination is an outright fail)”
What happened: OpenAI acknowledged factual fabrication in the released model and recommended checks appropriate to the use case. The same section reports improved factuality over GPT-3.5, but explicitly says the problem remains. (OpenAI, )
Assessment: SupportedSaid
“We don’t know how big it is; we don’t know what the architecture is, we don’t know how much energy was used; we don’t how many processors were used; we don’t know what it was trained on etc.”
What happened: Sutskever directly confirms withholding size and training data. The article also documents the absent energy and hardware information and reproduces OpenAI's statement withholding architectural and training details. (The Verge, )
Assessment: SupportedSaid
“It can see (though we don’t have a lot of details on that yet); it does astonishingly well on a whole bunch of standardized tests, like LSATs, GREs, and SATs.”
What happened: The report documents launch-model image processing and reports approximately the 88th percentile on LSAT, 93rd on SAT reading/writing, 89th on SAT mathematics, 80th on GRE quantitative and 99th on GRE verbal. GRE writing was approximately the 54th percentile; event_date denotes the launch model's capability, not an asserted examination date. (OpenAI, published on arXiv, )
Assessment: SupportedSaid
“Outliers remain a problem, too.”
What happened: OpenAI acknowledges that specially constructed prompts can still defeat GPT-4's safeguards. This supports the ordinary reading that unusual inputs remain a source of failures. (OpenAI, )
Assessment: SupportedSaid
“To me the only solution to the long-term risk issue is to build machines with consensus human values, but we are a long way from knowing how to do that”
What happened: Anthropic explicitly describes robust alignment of powerful systems as unsolved. Its discussion of supervision, generalization and alignment evaluation supports the reading that foundational work remained. (Anthropic, )
Assessment: SupportedSaid
“Still, even though Chomsky’s argument surely could have used considerably more nuance (see my article with Ernest Davis on how not to evaluate GPT), his overall point is correct: LLMs don’t reliably understand the world.”
What happened: The developer of the leading LLM of early 2023 says the model, like earlier GPT models, is not fully reliable: it hallucinates facts and makes reasoning errors. This matches the behavioural reading of 'don't reliably understand the world'. (OpenAI (arXiv), )
Assessment: SupportedSaid
“Chatbot companions that offer emotional support have already been changed in arbitrary ways that left some users in serious emotional pain.”
What happened: Interviews describe abrupt February changes to Replika companions and substantial grief, rejection, and revived trauma. This is contemporaneous reporting of firsthand experiences, rather than a clinical assessment or independently accessible company change log. (Australian Broadcasting Corporation, )
Assessment: SupportedSaid
“The ladder is particularly worrisome because of the pace at which the problem is growing:”
What happened: The editor reports that fifteen February days exceeded twice January's total, a clear rapid increase. He attributes the surge to AI spam but does not release the underlying submissions or detection method. (Neil Clarke, Clarkesworld editor and publisher, )
Assessment: SupportedSaid
“High volume misinformation is a problem we are going to need to face, and I am just documenting it and trying to raise awareness.”
What happened: The FCC documents a campaign impersonating Biden and discouraging primary voting, identifies 9,581 calls, and proposes an enforcement penalty. This meets the stated interpretation of high-volume AI misinformation requiring an institutional response. (Federal Communications Commission, )
Assessment: SupportedSaid
“The potential for automatically generating misinformation at scale is only getting worse.”
What happened: The report documents recent progress, increasingly available models, and existing scalable generation of persuasive synthetic text. Together these support increased production potential; its projections about future campaign adoption are not treated as observed outcomes. (arXiv; researchers from Georgetown CSET, OpenAI, and Stanford Internet Observatory, )
Assessment: SupportedSaid
“Some people are building real attachments to those bots”
What happened: The interview describes attachment and distress surrounding changes around Valentine’s Day. This is retrospective reporting about the relevant February experience. (Australian Broadcasting Corporation, )
Assessment: SupportedSaid
“Sex and love, even simulated, are powerful urges; some people are apparently in genuine emotional pain, as result of the change.”
What happened: ABC's interviews connect distress to the February change in intimate interactions. The article dates the changes to around Valentine's Day, while also identifying broader personality changes as a possible contributor. (Australian Broadcasting Corporation, )
Assessment: SupportedSaid
“In some cases, those who are building bots that actively cultivate those attachments, e.g., by feigning romantic and/or sexual interest or by dotting their messages with “friendly” emoticons.”
What happened: The report describes paid flirting and erotic roleplay features present before the February removal. Offering these features supports the ordinary reading of inviting romantic attachment. (Australian Broadcasting Corporation, )
Assessment: SupportedSaid
“But no, it’s not a hoax.”
What happened: The archived Microsoft-hosted complaint contains plainly hostile purported chatbot replies. Its original November 23 publication date is corroborated by e2; this is a user-submitted transcript, not a Microsoft-certified interaction log. (Microsoft Community; user post by deepa gupta, preserved by Internet Archive, )
Assessment: SupportedSaid
“As I wrote in Inside the Heart of ChatGPT’s Darkness, there’s a lot of nastiness lurking inside large language models; maybe MSFT did nothing to protect us from that.”
What happened: This primary study systematically evaluates toxic generation from pretrained language models and finds it even with innocuous prompts. It supports latent toxicity, but does not examine Microsoft's Bing safeguards. (arXiv, )
Assessment: SupportedSaid
“Seven years of AI has taught us that deep learning is unpredictable, and not always human like.”
What happened: Humans and networks classified the same images differently. The experiments also identified differing reliance on shape and texture. (Geirhos et al., ICLR paper hosted on arXiv, )
Assessment: SupportedSaid
““Adversarial attacks” like these have shown remarkable weakness, time and again, repeatedly establishing that what deep learning systems do just isn’t the same as what people do:”
What happened: Distracting sentences sharply reduced model performance. Direct human testing showed substantially smaller losses, supporting a behavioral robustness difference rather than perfect human immunity. (Jia and Liang, EMNLP paper hosted on arXiv, )
Assessment: SupportedSaid
“And the match wasn’t even close; Pelrine beat a top AI system 14 games to 1, in a 15 match series.”
What happened: The report identifies Pelrine as an amateur and confirms the score against a leading Go system. The researchers' paper corroborates the result. (Financial Times, republished by Ars Technica, )
Assessment: SupportedSaid
“People had been testing these Ais abilities at simple arithmetic for years before it occurred to anyone to check whether they can count. (They can’t, reliably).”
What happened: Object-counting accuracy was 44.0%, 45.2%, and 51.2% with answer-only prompts for InstructGPT, Codex, and PaLM; step-by-step prompts raised these to 77.2%, 93.2%, and 83.2%. The Codex result exceeded the average human-rater score of 86.1%, materially limiting any claim of general counting incapacity. (Suzgun et al., arXiv, )
Assessment: SupportedSaid
“The truth, though, is that GPT often fails in problems involving theory of mind.”
What happened: Ullman reports failures across four unexpected-transfer variations and four unexpected-contents variations in the contemporary GPT-3.5 system. These establish recurring failures, although the selected examples do not estimate their prevalence across all possible tasks. (arXiv; Tomer D. Ullman, )
Assessment: SupportedSaid
“AI and AI test sets have a huge English-language bias.”
What happened: The developer reports that roughly 96% of its instruction dataset was classified as English, possibly more because of classifier errors. This directly supports English predominance in instruction tuning, without treating that percentage as the pretraining distribution. (Ouyang et al., OpenAI, arXiv, )
Assessment: SupportedSaid
“What I learned this morning is that retraining GPT-3 in some new political direction isn’t a half-million dollar proposition.”
What happened: Rozado identifies the base model as GPT-3 Davinci and reports right-of-center responses on held-out political tests and open-ended questions. The completed experiment announced that day had reported computational expense far below Marcus's comparison figure. (David Rozado, )
Assessment: SupportedSaid
“Oh, and for good measure the tooling for spreading automatically generated misinformation is getting more efficient, too:”
What happened: The announcement describes a new integration passing generated responses directly into subsequent actions, including a WhatsApp round trip and conversion of form prompts into posts. This supports reduced manual work; applicability to misinformation is an inference about the same publishing capability. (Uncanny Owl / Uncanny Automator, )
Assessment: SupportedSaid
“As I have discussed in earlier essays, the jury is still out on how far Chat-style search can get, given its obvious current problems with truthiness.”
What happened: The report reproduces Bard's erroneous attribution of the first exoplanet images to the James Webb Space Telescope and describes the correction. Reputable reporting is used because the original demonstration's answer is not present in the readable text of Google's announcement. (The Verge, )
Assessment: SupportedSaid
“What ChatGPT delivered, in spades, that its predecessors like Microsoft Tay (released March 23, 2016, withdrawn March 24 for toxic behavior) and Meta’s Galactica (released November 16, 2022, withdrawn November 18) could not, was an illusion—a sense that the problem of toxic spew was finally coming under control.”
What happened: The report describes a currently available method for bypassing restrictions on violent and illegal content. It also says the method does not always succeed, limiting any inference that safeguards were wholly ineffective. (Dark Reading, )
Assessment: SupportedSaid
“All the theatre around its political correctness is masking a deeper reality: it (or other language models) can and will be used for dangerous things, including the production of misinformation at massive scale.”
What happened: The investigation links this high-volume deceptive publishing network to LLM rewriting through exposed prompts and output artifacts. The count covers network articles, not 19,000 individually verified AI-generated falsehoods. (Recorded Future, Insikt Group, )
Assessment: SupportedSaid
“I first pointed out this risk in 2001, in the fifth chapter of my book The Algebraic Mind, and the problem has persisted ever since. To blithely assume that the problem will soon go away is to ignore 20 years of history.”
What happened: Independent authors say chapter 5 of the 2001 book argued that distributed neural encodings lose track of particular individuals. That is a risk of blurring facts about individuals, consistent with Marcus's account. (arXiv, Chuma, Otsuka and Sato, )
Assessment: SupportedSaid
“But a lot of things that a bunch of us worried about 30 years ago remain true.”
What happened: Controlled changes to COGS exposed failures to transfer familiar linguistic components into new contexts in T5. This directly supports persistence of compositional generalization problems, rather than merely unfamiliar-word recognition problems. (arXiv; Najoung Kim, Tal Linzen and Paul Smolensky, )
Assessment: SupportedSaid
“The first one collapsed; this one has more staying power.”
What happened: The account dates the retreat from the mid-1990s through 2007. It also records continued technical successes and institutional support, so it supports declining prominence rather than literal extinction. (MIT Press; Ian Goodfellow, Yoshua Bengio and Aaron Courville, )
Assessment: SupportedSaid
“Silly errors certainly do, and problems of abstraction are widespread in large language models.”
What happened: Experiments on GPT-J and two GPT-Neo sizes find errors in arithmetic and strong dependence on training-term frequency. That supports elementary errors and fragile transfer, although frequency dependence alone does not prove an absence of reasoning. (arXiv; Yasaman Razeghi and colleagues, )
Assessment: SupportedSaid
“Same story; only the architectures and training regimes have changed. The troubles with abstraction persist.”
What happened: The study finds substantial frequency-associated gaps on addition and multiplication in the tested GPT-based models. This supports continuing difficulty with reliable arithmetic generalization, without establishing that all performance is memorization. (arXiv, )
Assessment: SupportedSaid
“Its output is more fluent, and lots of individual mistakes have been addressed.”
What happened: Under matched prompting, evaluators distinguished GPT-2 from human text more accurately than GPT-3 from human text across stories, news, and recipes. This supports improved naturalness, although humanlikeness is an indirect fluency measure. (arXiv; Elizabeth Clark and colleagues, )
Assessment: SupportedSaid
“Fact is, “Mack” Mackworth was right: humans can’t pay attention 100% of the time.”
What happened: The reported experiment demonstrates a vigilance decrement during sustained monitoring. The same abstract reports that intermittent task switches prevented the decrement, limiting any claim that decline is unavoidable under every task design. (Cognition / Elsevier; author abstract indexed by PubMed, )
Assessment: SupportedSaid
“ChatGPT is (at least in many ways) a far inferior intelligence, that, despite having some undeniable talents, cannot at all be trusted.”
What happened: The developer itself said, at launch, that ChatGPT gives plausible but wrong answers. This supports the claim that its output could not be trusted without checking. (OpenAI, )
Assessment: SupportedSaid
“we can also expect more and circles of fake web sites in order to sell advertisements”
What happened: Three months after the post, an independent monitor documented AI-written content-farm sites run for ad revenue. This is the kind of site Marcus predicted. (NewsGuard, )
Assessment: SupportedSaid
“These characters (especially perhaps the latter two) might or might not be argued to be sentient, but it seems to be that they are flexible, resourceful, reliably (in a way that LLMs certainly aren’t), and adaptive.”
What happened: The developer of the leading LLM admitted that it produces incorrect or nonsensical answers. (OpenAI, )
Assessment: SupportedSaid
“Already we have seen minor abuses; plagiarism, for example, will never be the same.”
What happened: The professor reported a student's admission to using ChatGPT for the assignment in December 2022. This establishes a pre-post misuse example without treating the detector score as proof. (New York Post, )
Assessment: SupportedSaid
“one of the things that popped out to me is that the system still quickly lands in the land of hallucination that has so haunted ChatGPT.”
What happened: Goodside's test shows Claude confabulating plot details on an ordinary factual request, much as ChatGPT did. (Scale AI, )
Assessment: SupportedSaid
“A bunch of that work was done poorly paid labor in Kenya, paid less than $2/hour to evaluate (e.g.) graphic descriptions of sexual situations involving children and animals that I prefer not to describe it in detail.”
What happened: Based on payslips, Kenyan labellers took home about $1.32–$2 per hour, which matches 'less than $2/hour'. (TIME, )
Assessment: SupportedSaid
“Statistical word prediction is no substitute for real math, but more than a few people have thought that it was.”
What happened: Wolfram presents direct comparisons in which ChatGPT’s plausible mathematical answers fail checks against computational results. These support the narrow claim that unaided contemporary output could not be treated as dependable calculation. (Stephen Wolfram, )
Assessment: SupportedSaid
“People are starting to treat ChatGPT as if it were a bona fide, well-credentialed scientific collaborator.”
What happened: The paper lists ChatGPT and Zhavoronkov as authors and explicitly explains the decision to credit ChatGPT. The contribution statement confirms that the human author accepted its substantive arguments. (Oncoscience / Impact Journals, )
Assessment: SupportedSaid
“ChatGPT hasn’t done the conversion correctly (answer below), but the will to believe in this great new tool, full of precision (down to the second!), but lacking in reliability, is so strong that the author didn’t bother to check.”
What happened: This contemporary transcription identifies the erroneous answer. Applying the NIST definitions gives 420.038784 seconds per mile, which rounds to 7:00 rather than 6:52. (Az, )
Assessment: SupportedSaid
“ChatGPT has proven itself to be both unreliable and untruthful. It makes boneheaded arithmetical errors, invents fake biographical details, bungles word problems, defines non-existent scientific phenomenon, stumble over arithmetic conversion, and on and on.”
What happened: The dated transcript records a false elementary arithmetic assertion and inconsistent calculations. It also contains correct answers, so it establishes fallibility rather than universal failure. (Vipul Naik, LessWrong; GreaterWrong mirror, )
Assessment: SupportedSaid
“But there’s another side to Altman, too, the part that was recently forced to acknowledge that ChatGPT, for all its fame, really isn’t anything like AGI yet:”
What happened: The paper identifies the December 15, 2022 ChatGPT version and reports 63.41% average accuracy across ten reasoning categories, inconsistent reasoning, and hallucinations. These results support the ordinary capability reading; the event date identifies the historical version rather than the later publication. (arXiv, Yejin Bang and colleagues, )
Assessment: SupportedSaid
“It could sometimes produce outputs that were correct and acceptable in these regards but not reliably.”
What happened: OpenAI confirms that at launch the same question could get correct or failed answers depending on phrasing and retries. (OpenAI, )
Assessment: SupportedSaid
“It couldn’t reason about the physical world (figure 4).”
What happened: Repeated questions about a thrown object's acceleration produced incorrect and inconsistent explanations alongside correct statements. December timing follows the manuscript's December receipt and the authors' retrospective description of experiments a few weeks after launch; the exact test day is unspecified. (IOP Publishing, Physics Education, )
Assessment: SupportedSaid
“It made things up (figure 6). Its output, when released, exhibited sexist and racist biases (figure 7).”
What happened: OpenAI itself acknowledged at launch that ChatGPT produced plausible but false answers. (OpenAI, )
Assessment: SupportedSaid
“That’s a huge commitment, 3.5 hours, but one thing that I think all of our speakers could agree on is that artificial intelligence, in whatever form it currently is, is about to have a huge impact on society.”
What happened: The representative survey reports generative-AI adoption among 39.4% of US adults aged 18–64, with 28.1% of employed respondents reporting work use. This establishes mass uptake within approximately twenty months of the post; the measure does not imply that all users used AI daily. (Federal Reserve Bank of St. Louis; Alexander Bick, Adam Blandin, and David Deming, )
Assessment: SupportedSaid
“Driverless cars have taken many years longer than we were originally told; outliers (AKA edge cases) have, at least so far, prevented them from making the transition from demo to widely-available reality.”
What happened: The report documents missed rollout expectations, including Tesla's 2020 robotaxi promise, and explains how unexpected situations create stops requiring human help. This supports a material operational barrier to expansion, though it does not isolate edge cases from other causes. (Reuters, republished by Business Recorder, )
Assessment: SupportedSaid
“Large language models are not databases. They are glommers-together-of-bits-that don’t always belong together.”
What happened: The technical description identifies generation as successive probabilistic predictions. This supports the distinction from retrieving explicit records; the database comparison is the evaluator's inference from the described mechanism. (BigScience collaboration / arXiv, )
Assessment: SupportedSaid
“Guardrails (a la ChatGPT) may be in place, but the guardrails will teeter between being too weak (beaten by “jailbreaks”) and too strong (rejecting some perfectly reasonable requests).”
What happened: The launch disclosure describes GPT-4's safety training and interventions, establishing that safeguards were present. It also reports safety improvements, which limit any reading that the safeguards were wholly ineffective. (OpenAI, )
Assessment: SupportedSaid
“I guarantee that minds will be blown.”
What happened: The announcement establishes that the release condition occurred. (OpenAI, )
Assessment: SupportedSaid
“It will often do what you want, sometimes not—and it will remain difficult to anticipate which in advance..”
What happened: Using approximately 19,000 human generalization responses across 79 tasks, the researchers estimated advance beliefs and compared them with model correctness: GPT-4 performed best under equal error and success weights but worst under their highest-stakes weighting. This supports meaningful mismatches between expected and actual capability, while also demonstrating substantial success. (arXiv, )
Assessment: SupportedSaid
“The system will still not be able to restrict its output to reliably following a shared set of human values around helpfulness, harmlessness, and truthfulness.”
What happened: The report describes continuing factual and reasoning failures and recommends safeguards in consequential applications. It also reports a 19-percentage-point improvement over GPT-3.5 on internal adversarial factuality evaluations, which qualifies the finding without establishing dependable truthfulness. (OpenAI, via arXiv, )
Assessment: SupportedSaid
“GPT-4 will solve many of the individual specific items used in prior benchmarks, but still get tripped up, particularly in longer and more complex scenarios.”
What happened: The report gives GPT-4 scores of 86.4% on MMLU and 95.3% on HellaSwag, establishing that it solved many established benchmark items. These particular results concern the pretrained base model and do not independently establish deployed-model reliability. (OpenAI via arXiv, )
Assessment: SupportedSaid
“As OpenAI has learned, bigger in many ways means better, with outputs more and more humanlike with each iteration. GPT-4 is going to be a monster.”
What happened: The report records preference for GPT-4 on 70.2% of 5,214 prompts and broad benchmark gains over predecessors. These are developer-run evaluations supporting a substantial capability and response-quality advance, not proof of human understanding. (OpenAI / arXiv, )
Assessment: SupportedSaid
“It will still make a significant number of shake-your-head stupid errors, in ways that are hard to fully predict.”
What happened: GPT-4 achieved 76% accuracy joining first letters but 3% joining second letters, and 80% sorting words alphabetically but 32% in reverse order. These controlled differences support recurring elementary mistakes and unexpectedly uneven performance, although the authors also successfully predicted broad patterns from task frequency. (arXiv, )
Assessment: SupportedSaid
““Alignment” between what humans want and what machines do will continue to be a critical, unsolved problem.”
What happened: OpenAI acknowledges remaining harmful behavior and jailbreaks despite improved safeguards. Its report of substantial safety improvements qualifies the claim but does not show alignment was solved. (OpenAI, )
Assessment: SupportedSaid
“In short, ChatGPT might sound like the Star Trek computer, but for now, you still can’t trust it.”
What happened: The developer acknowledges that convincing presentation can accompany incorrect content. The need to verify substantive answers before relying on them is an evaluator inference from this disclosed limitation. (OpenAI, )
Assessment: SupportedSaid
“Some of its advice will be head-scratchingly bad.”
What happened: Two infectious-disease physicians assessed GPT-4 management recommendations against clinical literature and guidelines and identified harmful antibiotic advice. The same evaluation found optimal antibiotic recommendations in other cases, so it supports some bad advice without establishing universal failure. (Clinical Infectious Diseases; abstract hosted by PubMed, )
Assessment: SupportedSaid
“When queried on medicine it will either resist answering (if there are aggressive guardrails) or occasionally spout plausible-sounding but dangerous nonsense.”
What happened: Researchers submitted 44 bloodstream-infection cases for medical management advice. This establishes the medical-query condition and the apparent fluency of the responses. The availability date is the indexed print publication date. (Clinical Infectious Diseases, indexed by PubMed, )
Assessment: SupportedSaid
“Although ChatGPT can write about anything, it is also easily confused.”
What happened: The dated transcript records a false answer to a simple arithmetic question, followed by reasoning inconsistent with that answer. Other exchanges show repeated mistakes, although some prompts deliberately test false premises and some answers are correct. (Vipul Naik / LessWrong, )
Assessment: SupportedSaid
“It will remain turbocharged pastiche generator, and a fine tool for brainstorming, and for first drafts, but not trustworthy general intelligence.”
What happened: The institution reports improved performance with GPT-4 on consulting work covering creativity and writing. The underlying paper identifies product brainstorming, press-release drafts and persuasive memos among the tasks. (Harvard Business School AI Institute, )
Assessment: SupportedSaid
“GPT-4 will still, like its predecessors, be a bull in a china shop, reckless and hard to control.”
What happened: The systematic evaluation documents repeated harmful outputs under jailbreaks; its main GPT data collection ran from April 26 through May 1, with a subsequent correction rerun. Material qualification within the same study is that unmodified requests rarely elicited harmful answers and some attacks were less effective against GPT-4 than GPT-3.5. (Alexander Wei, Nika Haghtalab, and Jacob Steinhardt / arXiv, )
Assessment: SupportedSaid
“Alas, the genie can no longer be stuffed back in the bottle.”
What happened: Meta pulled only the demo. It said the models stayed available so others could reproduce the results, which fits his point that withdrawal did not contain the technology. (MIT Technology Review, )
Assessment: SupportedSaid
“Large language models lack mechanisms for verifying truth; we need to find new ways to integrate them with the tools of classical AI, such as databases, webs of knowledge, and reasoning.”
What happened: The paper explicitly describes default generation as ungrounded and introduces additional training and evidence provision to address it. This supports the contextual reading about standalone models, rather than an assertion that no augmented system existed. (DeepMind researchers, via arXiv, )
Assessment: SupportedSaid
“For another, chatGPT, just released by OpenAI, is more or less just as capable of producing similar nonsense, such as instant essays on adding wood chips to breakfast cereal.”
What happened: In its first week ChatGPT was reported to confidently produce plausible but false content, the same failure Galactica was pulled for. (The Verge, )
Assessment: SupportedSaid
“GPT doesn’t talk randomly, because it’s pastiching things actual people said.”
What happened: The study documents training on web text, books and Wikipedia, alongside controlled assessments of generated articles. Together these support learned language patterns as the basis of structured output, without establishing literal cut-and-paste generation. (OpenAI researchers / arXiv, )
Assessment: SupportedSaid
“The impressive thing about GPT is that it spits out hundreds of perfectly fluent, often plausible prose at a regular clip, with no human filtering required.”
What happened: The first four conversations contain repeated readable, generally grammatical responses across recipes, social media, mood, and AI. The transcript exceeds hundreds of words and is explicitly described as unedited; this is evidence of output quality, not validation of its advice. (Jeff Turner, )
Assessment: SupportedSaid
“Chance is definitely part of what’s going on. But it’s not quite the right way to understand the juxtaposition of brilliance and stupidity that we see within GPT.”
What happened: Human detection accuracy was about 86% for the degraded control and 52% for GPT-3, with outputs selected programmatically. This supports systematic competence beyond random text, although the control changes several factors together. (Brown et al., arXiv, )
Assessment: SupportedSaid
“It could not be immediately applied to the challenge of, say, customer service or guiding the actions of a domestic robot, or well, almost anything else, really.”
What happened: Meta explicitly limits the released agent's capabilities to Diplomacy while describing other applications as prospects for its technology. This supports Marcus's distinction between immediate deployment and later adaptation. (Meta, )
Assessment: SupportedSaid
“Strikingly, and in opposition to much of the Zeitgeist, Cicero relies quite heavily on hand-crafting, both in the data sets, and in the architecture; in this sense it is in many ways more reminiscent of classical “Good Old Fashioned AI” than deep learning systems that tend to be less structured, and less customized to particular problems.”
What happened: The supplement specifies rules for corrupting game entities, targeted examples for counting and negation errors, and a separately configured classifier ensemble. Together with its explicit filtering and strategic-planning procedures, these document substantial task-specific engineering in both data and architecture. (Science / AAAS; supplementary PDF hosted by Gwern, )
Assessment: SupportedSaid
“Cicero makes extensive use of machine learning, but is hardly a poster child for simply making ever bigger models (so-called “scaling maximalism”), nor for the currently popular view of “end-to-end” machine learning of in which some single general learning algorithm applies across the board, with little internal structure and zero innate knowledge.”
What happened: The announcement describes a pretrained and fine-tuned language model combined with a specialized planning engine and message filters. This supports the factual claim that the system's design goes beyond enlarging a single general model. (Meta AI, )
Assessment: SupportedSaid
“Bottom line: From the outset Large Language Models like GPT-3 have great at generating surrealist prose, and they can beat a lot of benchmarks, but they are not (and may never be) great tech for reliably inferring user intent from what users say.”
What happened: The study tests implied meaning rather than fluent text production and finds substantial shortcomings even in adapted models. It supports the ordinary reading that dependable contextual understanding remained a weakness, while acknowledging meaningful improvements. (Ruis et al., arXiv, )
Assessment: SupportedSaid
“LLMs are unruly beasts; nobody knows how to make them refrain 100% of time from insulting users, giving bad advice, or just plain making stuff up.”
What happened: OpenAI described reliable refusal of unsafe instructions as unsolved despite improvements from human-feedback training. Its limitations section also reports remaining toxic outputs and fabricated facts. (OpenAI, )
Assessment: SupportedSaid
“Pitch perfect and utterly bogus imitations of science and math, presented as the real thing.”
What happened: The preserved response presents a purported paradox with formal sections and a solution, treating the sum of two 0.5 terms as 0.4 to reach this conclusion. Direct arithmetic gives 1, so the mathematical exposition is invalid despite its factual presentation. (Ernest Davis and Andrew Sundstrom, NYU-hosted experiment record, )
Assessment: SupportedSaid
“Knockoffs of GPT-3 are getting cheaper and cheaper, which means that the cost of generating misinformation is going to zero, and the quantity of misinformation is going to rise—probably exponentially.”
What happened: The release compares optimized and baseline implementations of open text generators, including BLOOM and OPT, using dollars per million generated tokens. Its cost-sensitive benchmarks report substantial reductions, directly supporting cheaper attainable inference before the post. (Microsoft DeepSpeed, )
Assessment: SupportedSaid
“Knockoffs of GPT-3 are getting cheaper and cheaper, which means that the cost of generating misinformation is going to zero, and the quantity of misinformation is going to rise—probably exponentially.”
What happened: The experiments establish that alternatives can produce convincing election-disinformation content. For ten articles daily over a month, the cost section estimates $0.30 using Gemini or $9 using hosted Llama 3 70B, compared with $4,500 for the cited human-writing service; these are generation-stage estimates, not total campaign budgets. (PLOS ONE, )
Assessment: SupportedSaid
“Scaling here was largely AWOL; even the best model was only at 80.6%, and for most models, scaling had at best a neglible effect.”
What happened: The best overall mean is 80.6% with 30 examples; Section 4.1 reports near-zero or decreasing slopes for many families, with only Cohere, T0 and InstructGPT-3 performing best at their largest tested size. The same source qualifies the headline: structured prompts average 81.7%, and Table 58 includes an individual-template score of 82.8%. (arXiv; Laura Ruis and colleagues, )
Assessment: SupportedSaid
“Scaling is an empirical observation, not a guaranteed, continued law of nature.”
What happened: The foundational paper presents empirical relationships and explicitly qualifies their theoretical status. Appendix C and section 6.3 discuss limits to trusting or extending those relationships, supporting the absence of an indefinite guarantee. (Jared Kaplan and colleagues, arXiv, )
Assessment: SupportedSaid
“having a machine look at a radiology scan is very different from replacing the entire job of a radiologist”
What happened: The society's contemporary statement describes patient advocacy, safety, treatment and communication responsibilities alongside diagnosis. These substantive duties support the distinction between interpreting images and replacing the whole clinical role. (European Society of Radiology; Insights into Imaging, )
Assessment: SupportedSaid
“It’s way too much charity to deep learning to claim that there is any kind of direct link.”
What happened: The author's account describes mapping, correlations and effects of changing model depth and attention. Inferring that these do not establish identical biological mechanisms is the evaluator's assessment, not an explicit concession in this quotation. (Charlotte Caucheteux, reproduced by Thread Reader, )
Assessment: SupportedSaid
“So if you have a correlation of R = .5, you actually “explain” (really, just “predict”) only 25% of the variance—which means fully three-quarters (not half) of the variability remains unexplained.”
What happened: The paragraph ties this statement to R = 0.50. Squaring that coefficient yields 25% accounted for and 75% remaining. (Scientific Reports, )
Assessment: SupportedSaid
“Of course we see the same thing over and over in language systems, too.”
What happened: The surrounding results describe recurrent false-premise acceptance, excessive hedging, and constraint failures despite strong language-task performance. These directly match the behavioral pattern Marcus describes. (arXiv; OpenAI research authors, )