Claims checked against evidence by Gary Marcus, page 14 of 19
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Claims checked against evidence by Gary Marcus (1815)
Each of these claims was checked against dated evidence.
Page 14 of 19: claims 1301 to 1400 of 1815, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Observation
“A good system should give the user a manifest of sources; current systems don’t.”
Our reading: Current generative AI systems do not give users a manifest of sources.
No horizon Evidence as of
Limitation: The outcome depends on the contextual meaning of “manifest”.
Source: Things are about to get a lot worse for Generative AI
Assessment: SupportedSaid
Observation
“Generative AI systems like DALL-E and ChatGPT have been trained on copyrighted materials;”
Our reading: Generative AI systems such as DALL-E and ChatGPT were trained on copyrighted materials.
No horizon Evidence as of
Limitation: The records do not provide a complete inventory of training works or establish which material entered every model version.
Source: Things are about to get a lot worse for Generative AI
Assessment: UnresolvedSaid
Observation
“OpenAI’s image software (which we accessed through Bing) is perfectly capable of verbatim and near-verbatim repetition of sources as well.”
Our reading: OpenAI's image generator can reproduce source images verbatim or near-verbatim.
No horizon Evidence as of
Limitation: A broader reading that counts recognizable character designs as near-verbatim sources would require different criteria.
Source: Things are about to get a lot worse for Generative AI
Assessment: UnresolvedSaid
Observation
“They do not inform users when they do so.”
Our reading: Generative AI systems do not inform users when they produce infringing material.
No horizon Evidence as of
Limitation: The displayed completion cannot establish the absence of notices elsewhere in the interface [e1].
Source: Things are about to get a lot worse for Generative AI
Assessment: UnresolvedSaid
Observation
“The output was not marked as being directly premised on copyrighted work”
Our reading: Midjourney's output was not labelled as being directly based on copyrighted work.
No horizon Evidence as of
Limitation: Absence of labelling is hard to document from secondary sources.
Source: An artist fights back, and Midjourney has embarrassed themselves
Assessment: SupportedSaid
Observation
“It’s also hardly the first time AI is helped with mathematics.”
Our reading: FunSearch is not the first time AI has helped with mathematics.
No horizon Evidence as of
Limitation: This does not adjudicate the narrower claim that FunSearch was a first for LLM-based mathematical discovery.
Source: Sorry, but FunSearch probably isn’t a milestone in scientific discovery
Assessment: SupportedSaid
Observation
“There is no evidence that the problem was heretofore “unsolvable””
Our reading: There is no evidence that the problem FunSearch addressed was previously unsolvable.
No horizon Evidence as of
Limitation: Support is limited to the ordinary distinction between impossibility and an open problem.
Source: Sorry, but FunSearch probably isn’t a milestone in scientific discovery
Assessment: SupportedSaid
Observation
“Human mathematicians had to write problem-specific code for each separate mathematical problem.”
Our reading: FunSearch required human mathematicians to write problem-specific code for each separate problem.
No horizon Evidence as of
Limitation: The outcome depends on interpreting mathematicians as human task designers and separate problems as different task families.
Source: Sorry, but FunSearch probably isn’t a milestone in scientific discovery
Assessment: UnresolvedSaid
Observation
“But the big tech executives that would need to deal with it aren’t happy, and are doing their best after the fact to undermine a hard-won compromise that involved years of negotiations and multiple all-nighters.”
Our reading: Big tech executives are trying to undermine the newly negotiated EU AI Act compromise after the fact.
No horizon Evidence as of
Limitation: Public statements do not provide a complete record of private lobbying.
Source: Meta’s Chief AI Officer is lying about the EU AI Act
Assessment: ContradictedSaid
Observation
“It came in the middle of May 2022.”
Our reading: Foundation models (general purpose AI) were first mentioned in the EU AI Act process in mid-May 2022, before ChatGPT's release.
No horizon Evidence as of
Limitation: An earlier counterexample disproves the claimed date without establishing the absolute earliest mention.
Source: Meta’s Chief AI Officer is lying about the EU AI Act
Assessment: SupportedSaid
Observation
“The phrase “Added late in the text” — which tries to paint the regulation of foundation models as some kind of sneaky last minute move — is a massive, massive misrepresentation.”
Our reading: Claiming that foundation model regulation was added late to the EU AI Act text is a major misrepresentation.
No horizon Evidence as of
Limitation: The result depends on Marcus's explicit interpretation of late as last-minute.
Source: Meta’s Chief AI Officer is lying about the EU AI Act
Assessment: UnresolvedSaid
Observation
“What I proposed then, as a replacement to the Turing Test, was a Comprehension Challenge; even now, no software would be able to meet that challenge.”
Our reading: As of December 2023, no AI software could meet Marcus's Comprehension Challenge.
No horizon Evidence as of
Limitation: The original challenge lacks a scoring protocol.
Assessment: UnresolvedSaid
Observation
“Dubious Google Gemini videos aside, nobody is close to passing that yet.”
Our reading: As of December 2023, no one, Google's Gemini included, was close to passing the Comprehension Challenge.
No horizon Evidence as of
Limitation: The result depends materially on the unspecified meaning of close.
Assessment: SupportedSaid
Observation
“ELIZA (built in 1965-1966) beat GPT 3.5.”
Our reading: In the cited Turing Test study, the 1960s ELIZA chatbot outperformed GPT-3.5.
No horizon Evidence as of
Limitation: This establishes the reported sample ranking, not general superiority over GPT-3.5.
Assessment: SupportedSaid
Observation
“And, sorry Sam, when you actually look at the data. humans are still ahead.”
Our reading: When you actually look at the data in the cited Turing Test study, humans are still ahead of the AI systems tested.
No horizon Evidence as of
Limitation: The finding concerns this experiment and its prompts.
Assessment: SupportedSaid
Observation
“The whole project was quietly dropped a year or two later.”
Our reading: OpenAI quietly dropped its Rubik's cube robotics project a year or two after the demo.
No horizon Evidence as of
Limitation: The decisive company statement is reproduced in reputable news reporting; a standalone primary shutdown announcement was not retrieved.
Assessment: UnresolvedSaid
Forecast
“What we are going to see, in the fullness of time, I promise you, is that Gemini is more or less in the same ball park as GPT-4, handy for a bunch of things, but untethered in reality, still with dicey, unpredictable reasoning, and a very limited understanding of the world.”
Our reading: Gemini will be shown to still have dicey, unpredictable reasoning and a very limited understanding of the world.
No deadline stated Evidence as of
Limitation: Competitive benchmark performance does not prove broad world understanding.
Assessment: UnresolvedSaid
Observation
“Regular readers will recall how that turned out: the Rubik’s cube was instrumented with sensors, and not straight off the shelf, and what worked for the Rubik’s cube in the demo video didn’t turn out have a whole lot of application in the real world.”
Our reading: OpenAI's Rubik's cube robot demo relied on a sensor-instrumented cube and had little real-world application.
No horizon Evidence as of
Limitation: The evidence does not identify which demonstration video Marcus referenced.
Assessment: SupportedSaid
Observation
“Note that Gates and Altman have both been dropping hints, and GPT-5 isn’t here after a year despite immense commercial desire.”
Our reading: Gates and Altman have both been dropping hints, and GPT-5 has not arrived after a year despite immense commercial desire.
No horizon Evidence as of
Limitation: Support depends on treating the year as an approximate ChatGPT-era anniversary.
Assessment: UnresolvedSaid
Observation
“From a commercial standpoint GPT-4 is no longer unique.”
Our reading: Commercially, GPT-4 is no longer a unique offering.
No horizon Evidence as of
Limitation: The disclosures provide no matched evaluation establishing broad commercial comparability with GPT-4 on December 6 [e1, e2].
Assessment: SupportedSaid
Observation
“But that’s exactly what’s happening: Messrs. LeCun and Zuckerberg seem to have talked it out, and decided what’s best (in their not-so-humble opinion) for all of us.”
Our reading: Marcus says it is exactly what's happening that a couple of tech lords are deciding for us: LeCun and Zuckerberg seem to have talked out whether to open-source Meta's AI and decided what's best, in their opinion, for all of us.
No horizon Evidence as of
Limitation: Decision authority partly rests on retrospective reporting.
Source: Critical national security questions should not be decided unilaterally by unelected tech leaders
Assessment: UnresolvedSaid
Observation
“So far, Big Tech rather than the US government is calling the shots.”
Our reading: On sharing AI technology with other nations (e.g., OpenAI's G42 partnership), Big Tech rather than the US government is currently making the decisions.
No horizon Evidence as of
Limitation: Private diplomacy and agreement terms are incompletely public.
Source: Critical national security questions should not be decided unilaterally by unelected tech leaders
Assessment: UnresolvedSaid
Observation
“But virtually all existing laws were written before AI was really on the scene, and so very few really envision how AI changes things.”
Our reading: Virtually all existing laws were written before AI was really on the scene, and so very few of them really envision how AI changes things.
No horizon Evidence as of
Limitation: The cited evidence describes earlier enactments and is context for the publication-date legal stock.
Assessment: SupportedSaid
Observation
“Its pioneering, world-leading AI Act is on the ropes.”
Our reading: As of November 2023, the EU AI Act is in danger of failing.
No horizon Evidence as of
Limitation: The decisive source is reporting of participant interviews, not an official transcript or complete negotiating record.
Assessment: UnresolvedSaid
Observation
“Big tech wants the world to revolve around its new plaything, Generative AI (combined total revenue so far a few billion dollars, just a tiny tiny fraction of the economy), and to exempt it from any regulation.”
Our reading: Generative AI's combined total revenue so far is only a few billion dollars, a tiny fraction of the economy.
No horizon Evidence as of
Limitation: Enterprise estimates omit or differently classify consumer products and infrastructure.
Assessment: UnresolvedSaid
Observation
“Actually, there was one clear overall winner: Tegmark. As much as Ernie Davis and I utterly disagreed with Tegmark’s more technical map of the world/LLM world models paper a few weeks ago, pretty much every word he said this week about regulation was true.”
Our reading: Nearly everything Tegmark said that week about AI regulation was true.
No horizon Evidence as of
Limitation: The evaluable reading concerns factual substance; the broader endorsement also contains opinion.
Assessment: SupportedSaid
Forecast
“And I do mean “we”; what the EU decides will affect all of us.”
Our reading: The EU's decision on regulating generative AI will affect people everywhere, not just in the EU.
Settled before its deadline Evidence as of
Limitation: Support depends on reading all of us as worldwide relevance rather than a literal claim about every person.
Assessment: UnresolvedSaid
Observation
“Bing,powered by Dall-E 3, was an even bigger loser, drawing “Hinton vs LeCun vs Ng vs Tegmark, boxing” with four boxers not boxing, not one of them looking remotely like any of our heroes:”
Our reading: Bing powered by DALL-E 3, prompted to draw Hinton, LeCun, Ng and Tegmark boxing, produced four boxers not boxing and resembling none of the people named.
No horizon Evidence as of
Limitation: No visual inspection was possible.
Assessment: UnresolvedSaid
Forecast
“Eventually, we will be able to build machines that help us reason critically, overcome human bias, and objectively, the way that we have calculators that spit out just the facts, without confabulation.”
Our reading: Eventually, we will be able to build machines that help us reason critically, overcome human bias, and reason objectively, giving just the facts without confabulation, the way calculators do.
No deadline stated Evidence as of
Limitation: Small, task-specific studies cannot establish broad capability.
Source: AI versus The Clumsy Human Mind
Assessment: UnresolvedSaid
Observation
“But “breakthroughs” rarely turn to be general to live up to initial rosy expectations. Often advances work in some contexts, not otherwise.”
Our reading: Claimed AI breakthroughs rarely prove general enough to live up to initial expectations, often working only in some contexts.
No horizon Evidence as of
Limitation: The result depends on which advances and which expectations are counted.
Source: About that OpenAI “breakthrough”
Assessment: SupportedSaid
Observation
“getting their algorithm to work for one object (which, cheesily, turned out to be a special Cube instrumented with sensors and LEDS, unlike a Rubik’s cube you’d buy in a store) in carefully controlled lab circumstances hardly guaranteed that the solution would work more broadly in the complex open-ended real world.”
Our reading: OpenAI's 2019 Rubik's cube robot result, achieved on a sensor-instrumented cube in controlled lab conditions, did not guarantee generalization to the open-ended real world.
No horizon Evidence as of
Limitation: The selected documentation does not establish the precise claim that one demonstrated cube contained both sensors and LEDs.
Source: About that OpenAI “breakthrough”
Assessment: SupportedSaid
Observation
“AV’s still aren’t general enough that you can just plop down a car that was tuned in pilot studies in Menlo Park, SF and Arizona and expect it to drive gracefully and safely in Sicily or Mumbai.”
Our reading: Autonomous vehicles still are not general enough that you can just plop down a car tuned in pilot studies in Menlo Park, SF and Arizona and expect it to drive gracefully and safely in Sicily or Mumbai.
No horizon Evidence as of
Limitation: The evidence documents Waymo's own testing process; it does not establish every developer's capabilities [e1].
Source: About that OpenAI “breakthrough”
Assessment: UnresolvedSaid
Causal claim
“It autocompletes sentences, some of those completions are true given that the training data is so vast, and some aren’t, given that the training data aren’t infinite.”
Our reading: GPT autocompletes sentences; some outputs are true because training data is vast and some are false because training data is not infinite.
No horizon Evidence as of
Limitation: This assessment does not interpret finite data as the sole asserted cause.
Source: A Tale of Two Liars
Assessment: SupportedSaid
Observation
“It’s fun to compare the two—they have so many differences—even though neither can be trusted.”
Our reading: GPT cannot be trusted.
No horizon Evidence as of
Limitation: The conclusion concerns unchecked factual reliance; the summarization results do not establish that every GPT answer is false or that GPT has no useful applications [e6].
Source: A Tale of Two Liars
Assessment: UnresolvedSaid
Observation
“The real value was always the people, not the IP, not the data, not the customer list, not the infrastructure.”
Our reading: OpenAI's real value was always its people, not its IP, data, customer list or infrastructure.
No horizon Evidence as of
Limitation: Workforce and technology value may depend on each other.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: UnresolvedSaid
Causal claim
“The whole nonprofit-hybrid model, introduced in 2019 (the company was originally pure non-profit) was always a ticking time bomb, because the people trying to make money were never going to have a perfect relationship with those in the nonprofit that were charged with keeping the world safe.”
Our reading: OpenAI's nonprofit-hybrid structure was a ticking time bomb because profit-seekers and safety-focused nonprofit stewards would inevitably conflict.
No horizon Evidence as of
Limitation: Both principal causal accounts come from parties involved in the dispute.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: SupportedSaid
Observation
“OpenAI’s strength has been in scaling up an existing paradigm (Transformers) that was developed elsewhere (Google), not developing new paradigms.”
Our reading: OpenAI's strength has been scaling the Transformer paradigm developed at Google, not developing new paradigms.
No horizon Evidence as of
Limitation: This is a qualitative assessment of the principal GPT advances, not a ranking of every OpenAI research contribution.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: SupportedSaid
Observation
“And the overriding nonprofit mission was always going to be at odds with a pure for profit play.”
Our reading: OpenAI's nonprofit mission was inherently at odds with a pure for-profit business.
No horizon Evidence as of
Limitation: The outcome depends on reading at odds as a structural constraint, not perpetual operational conflict.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: SupportedSaid
Observation
“There has never been a solution to the hallucination problem, and it’s not clear that there will be until we find a new paradigm, as Altman himself partly acknowledged last week, when he did his best Gary Marcus impression.”
Our reading: The hallucination problem has never been solved and may not be until a new paradigm is found.
No horizon Evidence as of
Limitation: Support concerns a general solution, not whether particular applications could achieve useful reliability.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: UnresolvedSaid
Forecast
“Large language models alone were never going to solve the alignment problem.”
Our reading: Large language models alone were never going to solve the alignment problem.
No deadline stated Evidence as of
Limitation: The experiments do not establish reliable alignment across all relevant deployment settings.
Source: Top 5 reasons why OpenAI was probably never really worth $86 billion
Assessment: SupportedSaid
Observation
“and then, perhaps in keeping with my surmise that Sam’s bluff was called, negotiations broke down altogether.”
Our reading: Negotiations (over Sam Altman's return to OpenAI) broke down altogether, which Marcus says is perhaps in keeping with his surmise that Altman's bluff was called.
No horizon Evidence as of
Limitation: The assessment depends on reading “altogether” within the post's immediate time frame.
Source: Everybody’s out
Assessment: SupportedSaid
Observation
“We pretty obviously don’t have a say.”
Our reading: The public has no say in how OpenAI's governance crisis is resolved, including its implications for safety and humanity.
No horizon Evidence as of
Limitation: The result depends on interpreting “a say” as formal participation.
Assessment: MixedSaid
Observation
“The spirit of the original arrangement was that everything that the for-profit did was supposed to be in the service of the non-profit. A board with no financial interest was supposed to look out for humanity.”
Our reading: OpenAI's original structure was meant to make the for-profit serve the non-profit, with a financially disinterested board looking out for humanity.
No horizon Evidence as of
Limitation: This assesses the intended structure, not its effectiveness.
Assessment: SupportedSaid
Forecast
“All signs are that those financially-interested stakeholders will quickly emerge victorious. (Arguably, they already have).”
Our reading: All signs are that OpenAI's financially interested stakeholders (employees and investors) will quickly win out over the nonprofit board, and arguably they already have.
Settled before its deadline Evidence as of
Limitation: Marcus supplied no exact deadline for 'quickly'.
Assessment: SupportedSaid
Observation
“In reality, things have gone quite differently. When the nonprofit Board felt it was prudent (for undisclosed reasons) to remove Altman, per the goals of the nonprofit that they were trying to represent, at the risk of potentially devastating commercial potential, those with a stake in the nominally subordinate for-profit (both employees and investors) quickly set to work to push out the board and to undo its decisions.”
Our reading: After OpenAI's nonprofit board removed Altman, employees and investors in the for-profit quickly worked to push out the board and undo its decision, contrary to the intended structure.
No horizon Evidence as of
Limitation: News reporting is used because the private negotiations lack accessible public primary records.
Assessment: SupportedSaid
Observation
“The fact that the governance of one of the most visible AI companies in the world can change literally overnight”
Our reading: OpenAI, one of the most visible AI companies, had its governance change literally overnight in November 2023.
No horizon Evidence as of
Limitation: This establishes a change in governing personnel, not a replacement of OpenAI's formal governance structure.
Source: The biggest immediate lesson from the OpenAI meltdown
Assessment: SupportedSaid
Observation
“Breaking news: the draft EU AI Act—the closest thing we have to AI regulation with real teeth—is in serious jeopardy, potentially a direct victim of regulatory capture, governments doing the bidding of those they are supposed govern.”
Our reading: The draft EU AI Act is in serious jeopardy as of November 2023.
No horizon Evidence as of
Limitation: No official minutes of the 10 November meeting were obtained; the assessment relies on reporting and retrospective legal analysis.
Source: Crisis point in AI?
Assessment: UnresolvedSaid
Forecast
“Driver-assist (a la Tesla) with humans in the loop may well stay, occasionally causing fatal accidents, given its lack of reliability, but the idea of using your car moonlighting as a self-driving taxi is not happening this decade, maybe not even next.”
Our reading: Personal cars moonlighting as self-driving taxis will not happen this decade, and maybe not next decade either.
Not yet due Evidence as of
Limitation: The packet does not establish exhaustive worldwide non-occurrence.
Source: Rethinking “driverless cars”
Assessment: UnresolvedSaid
Observation
“Essentially NONE of the comparisons we have heard between humans and machines have looked at cars in which humans were not playing some role in the driving, either in the car or remotely.”
Our reading: Essentially none of the human-versus-machine comparisons we have heard looked at cars in which humans played no role in the driving, either in the car or remotely.
No horizon Evidence as of
Limitation: The packet is not a comprehensive inventory of comparisons available by November 5, 2023.
Source: Rethinking “driverless cars”
Assessment: UnresolvedSaid
Forecast
“Uber and Lyft can stop worrying about being disintermediated by machines; they will still need human drivers for quite some time.”
Our reading: Uber and Lyft can stop worrying about being disintermediated by machines, because they will still need human drivers for quite some time.
No deadline stated Evidence as of
Limitation: Company descriptions of strategy are not independent measurements of future dependence.
Source: Rethinking “driverless cars”
Assessment: UnresolvedSaid
Forecast
“High-reliability is not their forte. and nor are outliers. They won’t take humans out of the loop.”
Our reading: LLMs (transformers) are not highly reliable and do not handle outliers well, and they won't take humans out of the loop in the pursuit of Level 5 self-driving.
No deadline stated Evidence as of
Limitation: Both items are company disclosures, not independent validation of qualifying Level 5 operation.
Source: Rethinking “driverless cars”
Assessment: SupportedSaid
Observation
“Update: A few hours after I posted this Cruise CEO Kyle Vogt essentially confirmed that their “driverless” cars need very regular human intervention:”
Our reading: Cruise's CEO essentially confirmed that Cruise's driverless cars require very regular human intervention.
No horizon Evidence as of
Limitation: Supported for recurring operational assistance; strict necessity for safety is not established.
Assessment: SupportedSaid
Forecast
“Shareholders will certainly sue, and if it’s bad as it looks, I doubt that GM will continue the project, which was recently suspended.”
Our reading: Shareholders will sue over Cruise.
Settled before its deadline Evidence as of
Limitation: This assessment concerns the lawsuit prediction only.
Assessment: ContradictedSaid
Observation
“I realize there’s been almost zero public disclosure of how any of the driverless car companies are using remote operators. It’s not just Cruise.”
Our reading: Driverless car companies in general have disclosed almost nothing publicly about their use of remote operators.
No horizon Evidence as of
Limitation: The result depends on reading the claim as about disclosure of operators’ roles, rather than complete quantitative transparency.
Assessment: UnresolvedSaid
Causal claim
“Certainly no commercial product could ever work at a profit if you needed remote operators anything like that often.”
Our reading: Certainly no commercial product could ever work at a profit if it needed remote operators anything like as often as Cruise reportedly does.
No horizon Evidence as of
Only if: if remote operators were needed anything like as often as reported
Assessment: SupportedSaid
Observation
“Nobody knows for SURE that there are no serious possible consequences of open source AI, e.g in terms of bioweapons, misinformation generation etc, as Séb Krier at DeepMind nicely highlights above.”
Our reading: No one knows for certain that open source AI has no serious possible consequences, such as for bioweapons or misinformation generation.
No horizon Evidence as of
Limitation: The experiments concern public Llama 2-Chat weights and do not establish equivalent results for every open model or for access through a controlled service [e1].
Source: Where we are right now on open source and potential AI risk.
Assessment: UnresolvedSaid
Causal claim
“How effective this actually becomes depends a lot on the exact wording, and how things are enforced, and how much of it is binding as opposed to merely voluntary.”
Our reading: The Executive Order's effectiveness depends heavily on exact wording, enforcement, and how much is binding rather than voluntary.
No horizon Evidence as of
Limitation: The available evaluations cover particular activities rather than the order's overall effects.
Assessment: UnresolvedSaid
Forecast
“Companies will poke for loopholes and surely find them.”
Our reading: AI companies will look for and find loopholes in the Executive Order.
No deadline stated Evidence as of
Limitation: Company compliance deliberations may not be public.
Assessment: UnresolvedSaid
Forecast
“We will eventually see search engines that return neatly written paragraphs, and eventually see AI agents that can organize lives. Or at least our grandchildren or their children will.”
Our reading: Search engines that return well-written paragraphs and AI agents that can organize lives will eventually arrive, though possibly only for our grandchildren or their children.
No deadline stated Evidence as of
Limitation: Pew did not directly grade the summaries' writing quality or factual reliability [e1].
Source: Why driverless cars might—or might not—be at the end of the road
Assessment: UnresolvedSaid
Causal claim
“To get there, though, what we need is a much smarter AI, a kind that can reason, and not hallucinate, grounded in reality and not just corpus statistics— which require fundamental research of a sort that is currently getting starved out in the LLM euphoria.”
Our reading: Getting to reliable self-driving cars that are safer than humans requires a much smarter AI that can reason, does not hallucinate, and is grounded in reality rather than just corpus statistics, and building that AI requires fundamental research.
No horizon Evidence as of
Limitation: The key capabilities and required research advances lack precise definitions.
Source: Why driverless cars might—or might not—be at the end of the road
Assessment: SupportedSaid
Forecast
“I—truly a techno-optimist, popular perceptions to the contrary—do think driverless will come, eventually.”
Our reading: Driverless cars will eventually arrive.
Settled before its deadline Evidence as of
Limitation: Marcus supplied no explicit adoption threshold, so interpretation materially affects the result.
Source: Why driverless cars might—or might not—be at the end of the road
Assessment: SupportedSaid
Causal claim
“you don’t have to be superintelligent to create serious problems.The real issue is control.”
Our reading: AI need not be superintelligent to create serious problems; the real issue is control.
No horizon Evidence as of
Limitation: The example concerns automated driving, not an LLM.
Assessment: SupportedSaid
Observation
“We have no way of guaranteeing that large language models will reliably stick to facts, a point so thoroughly established by now I don’t think it needs a reference.”
Our reading: There is no way to guarantee that large language models reliably stick to facts.
No horizon Evidence as of
Limitation: Support rests on a contemporary research synthesis, not an exhaustive proof of nonexistence.
Assessment: UnresolvedSaid
Observation
“Current AI has no way to reliably anticipate consequences of actions that it might recommend, Asimov’s First Law of Robotics called for robots to do no harm, but current AI has no reliable way to anticipate harm.”
Our reading: Current AI cannot reliably anticipate the consequences or harms of actions it recommends.
No horizon Evidence as of
Limitation: The scope of current AI and the meaning of reliably are unspecified.
Assessment: SupportedSaid
Observation
“We have no way of guaranteeing that large language models won’t be jailbroken into saying unethical things. To the contrary, a just-published technique shows that current LLMs can be jailbroken quickly, and automatically.”
Our reading: There is no way to guarantee LLMs won't be jailbroken, and current LLMs can be jailbroken quickly and automatically.
No horizon Evidence as of
Limitation: The guarantee clause depends on the stated general-immunity interpretation.
Assessment: SupportedSaid
Observation
“But, crucially, it often gets these wrong, particularly when they involve spatial reasoning beyond a cookie-cutter formula or reasoning about sequences of disparate events.”
Our reading: GPT-4 with either plug-in often gets wrong problems that a human solver would need significant thought to translate into math notation, particularly those involving spatial reasoning beyond a cookie-cutter formula or reasoning about sequences of disparate events.
No horizon Evidence as of
Limitation: The selected problems do not establish a general error rate for translation-intensive tasks or a measured comparison with human difficulty [e1].
Source: Getting GPT to work with external tools is harder than you think
Assessment: UnresolvedSaid
Observation
“In a few cases GPT-4 set up the calculations correctly and the plug-in returned the correct answer, but then GPT-4 either misunderstood or refused to accept that answer.”
Our reading: In a few test cases GPT-4 set up the calculations correctly and the plug-in returned the correct answer, but GPT-4 then either misunderstood or refused to accept that answer.
No horizon Evidence as of
Limitation: A 2024 transcript revision supplies important qualifications about the 2023 tests.
Source: Getting GPT to work with external tools is harder than you think
Assessment: SupportedSaid
Observation
“Second, GPT-4 doesn’t reliably understand how to use the tools.”
Our reading: GPT-4 does not reliably understand how to use the Code Interpreter and Wolfram Alpha plug-ins.
No horizon Evidence as of
Limitation: The main evidence comes from the study underlying the post, rather than an independent replication.
Source: Getting GPT to work with external tools is harder than you think
Assessment: MixedSaid
Observation
“In many cases, a naive user might take the results, invariably presented as fact, as correct, when they are not.”
Our reading: GPT-4 with plug-ins invariably presents its results as fact, and in many cases its results are incorrect even though a naive user might take them as correct.
No horizon Evidence as of
Limitation: The mixed result depends on reading invariably literally.
Source: Getting GPT to work with external tools is harder than you think
Assessment: SupportedSaid
Observation
“Finding a way to get LLMs to reliably use external tools is far from trivial.”
Our reading: Getting LLMs to reliably use external tools is far from trivial.
No horizon Evidence as of
Limitation: The conclusion uses an ordinary qualitative reading of difficulty.
Source: Getting GPT to work with external tools is harder than you think
Assessment: UnresolvedSaid
Observation
“Thus far we have seen consistently unreliable results, from math to fact-checking.”
Our reading: So far, the authors have seen consistently unreliable results from LLMs using external tools, from math to fact-checking.
No horizon Evidence as of
Limitation: The outcome depends on retaining the explicit personal scope, 'we have seen'.
Source: Getting GPT to work with external tools is harder than you think
Assessment: SupportedSaid
Causal claim
“Certainly, adding these plug-ins does improve things.”
Our reading: Adding the Code Interpreter and Wolfram Alpha plug-ins improves GPT-4's math performance.
No horizon Evidence as of
Limitation: The shared test set comprises difficult competition problems.
Source: Getting GPT to work with external tools is harder than you think
Assessment: SupportedSaid
Causal claim
“There is no question that the two plug-ins significantly extend GPT-4’s ability to solve math and science problems.”
Our reading: The Code Interpreter and Wolfram Alpha plug-ins significantly extend GPT-4's ability to solve math and science problems.
No horizon Evidence as of
Limitation: Product and hidden model settings were not fully controlled, limiting precise causal attribution [e1, e2].
Source: Getting GPT to work with external tools is harder than you think
Assessment: UnresolvedSaid
Observation
“First, neither GPT-4 nor the plug-ins have the commonsense world knowledge and reasonable abilities needed to reliably translate a word problem into a mathematical calculation.”
Our reading: Neither GPT-4 nor its plug-ins have the commonsense knowledge and reasoning needed to reliably translate word problems into calculations.
No horizon Evidence as of
Limitation: The cited example comes from a college electromagnetism task.
Source: Getting GPT to work with external tools is harder than you think
Assessment: SupportedSaid
Observation
“GPT-4 and similar current language-based AI programs notoriously struggle with mathematical calculations.”
Our reading: GPT-4 and similar current language-based AI programs struggle with mathematical calculations.
No horizon Evidence as of
Limitation: Results depend on operand sizes, operation mixtures, and prompting.
Source: Getting GPT to work with external tools is harder than you think
Assessment: SupportedSaid
Observation
“On some problems that capable students might well find hard, the AI did perfectly; on other problems that lot of mathematically-inclined college students people would find easy, the AI failed miserably.”
Our reading: In the authors' tests, GPT-4 with plug-ins did perfectly on some problems that capable students might well find hard, and failed miserably on other problems that many mathematically inclined college students would find easy.
No horizon Evidence as of
Limitation: No student comparison group was tested, so the evidence does not establish student success rates or a measured ranking against students [e1].
Source: Getting GPT to work with external tools is harder than you think
Assessment: UnresolvedSaid
Observation
“To our knowledge, no LLM-based system can yet reliably do what SHRLDU tried to do then:”
Our reading: As far as the authors know, no LLM-based system can yet reliably do the blocks-world modeling that SHRDLU attempted in the 1970s.
No horizon Evidence as of
Limitation: Planning benchmarks cover only part of the ordinary SHRDLU comparison.
Source: Hello, Multimodal Hallucinations
Assessment: SupportedSaid
Observation
“As is by now widely known, they are a hallmark of systems that generalize based on word distribution.”
Our reading: As is by now widely known, hallucinations are a hallmark of systems that generalize based on word distribution.
No horizon Evidence as of
Limitation: This establishes neither a single hallucination rate nor that all models perform equally poorly.
Source: Hello, Multimodal Hallucinations
Assessment: UnresolvedSaid
Forecast
“And no matter what investors and CEOs might tell you, hallucinations will remain inevitable.”
Our reading: Hallucinations in large language models will remain inevitable.
No deadline stated Evidence as of
Limitation: A mathematical reading and a practical-risk reading could yield different assessments.
Source: Hello, Multimodal Hallucinations
Assessment: SupportedSaid
Observation
“To be sure, large language models are, in a lots of ways, an incredible advance over the AI of the 1970s. They can engage in a much broader range of tasks, and generally they need less hand-tinkering (Though they still need some – that’s what reinforcement learning with human feedback is really about.)”
Our reading: In a lot of ways, large language models are an incredible advance over the AI of the 1970s: they can engage in a much broader range of tasks and generally need less hand-tinkering, though they still need some, which is what reinforcement learning with human feedback is really about.
No horizon Evidence as of
Limitation: The conclusion depends on reading 'hand-tinkering' as task-specific customization.
Source: Hello, Multimodal Hallucinations
Assessment: UnresolvedSaid
Causal claim
“But in other ways, large language models are a profound step backwards; they lack stable models of the world, which means that they are poor at planning, and unable to reason reliably; everything becomes hit or miss..”
Our reading: LLMs lack stable world models, which makes them poor at planning and unable to reason reliably.
No horizon Evidence as of
Limitation: Evidence supporting the behavioral component is not sufficient to score the entire causal proposition supported.
Source: Hello, Multimodal Hallucinations
Assessment: UnresolvedSaid
Observation
“ChatGPT went utterly wild when Ernie pushed on the last one, omitting some that are there and and confabulating a standalone island in the middle of the kitchen that simply isn’t there:”
Our reading: When questioned about the kitchen image, ChatGPT's vision model left out drawers that are there and made up a kitchen island that is not there.
No horizon Evidence as of
Limitation: The page reader does not reproduce the relevant image and transcript screenshots.
Source: Hello, Multimodal Hallucinations
Assessment: SupportedSaid
Observation
“As we can see, multimodal training is not magically making hallucinations go away.”
Our reading: Multimodal training is not making hallucinations go away.
No horizon Evidence as of
Limitation: The evidence does not isolate the causal effect of multimodal training or establish that it cannot reduce hallucination rates.
Source: Hello, Multimodal Hallucinations
Assessment: UnresolvedSaid
Observation
“The LLM never induces such an algorithm.”
Our reading: The LLM does not induce an algorithm that actually computes multiplication.
No horizon Evidence as of
Limitation: This does not establish that MathGLM learned a general multiplication algorithm.
Source: “Math is hard” — if you are an LLM – and why that matters
Assessment: UnresolvedSaid
Observation
“The first author’s nine-year-old can do five digit multiplication better than an LLM.”
Our reading: The first author's own nine-year-old child can do five-digit multiplication better than an LLM.
No horizon Evidence as of
Limitation: A private comparison may have occurred.
Assessment: UnresolvedSaid
Observation
“There are some applications where the current systems are reliable enough to be practically useful, though not nearly as many as is often claimed.”
Our reading: Current systems are reliable enough to be useful in some applications, but far fewer than often claimed.
No horizon Evidence as of
Limitation: The existence of useful assistance is easier to test than the claimed extent of overstatement.
Assessment: SupportedSaid
Observation
“Transformers and generative AI are getting a lot of attention, and have considerable utility, but they are nothing like the all purpose tools some people seems to imagine them to be.”
Our reading: Transformers and generative AI are getting a lot of attention and have considerable utility, but they are nothing like the all-purpose tools some people seem to imagine them to be.
No horizon Evidence as of
Limitation: The result depends on interpreting all-purpose as broadly reliable practical competence.
Assessment: SupportedSaid
Forecast
“No doubt the next generation of systems will have greater abilities and more extensive applications and will somewhat mitigate the weaknesses.”
Our reading: The next generation of systems will, without doubt, have greater abilities and more extensive applications and will somewhat mitigate the weaknesses of current systems.
Settled before its deadline Evidence as of
Limitation: The evidence is concentrated in the Claude successor family.
Assessment: UnresolvedSaid
Causal claim
“About 18 months have passed; although there are certainly improvements (the images are vastly better), owing presumably in large part to a larger training set, the fundamental approach (mapping words to images, rather than constructing intermediate models of the world and how language and images relate to those models) remains the same, and so the core issue remains.”
Our reading: About 18 months after DALL-E 2, the images are vastly better, presumably largely because of a larger training set, but the fundamental approach of mapping words to images rather than building intermediate world models is the same, and so the core compositionality issue remains.
No horizon Evidence as of
Limitation: The technical paper was read through a third-party reproduction because the original PDF exceeded the reader's size limit.
Source: Race, statistics, and the persistent cognitive limitations of DALL-E
Assessment: UnresolvedSaid
Causal claim
“It’s the same issue of canonical versus noncanonical statistics, all over again.”
Our reading: Multimodal ChatGPT's trouble telling time from watch images comes from the same canonical-versus-noncanonical statistics problem seen in DALL-E.
No horizon Evidence as of
Limitation: Later model evaluations cannot be substituted for the historical deployment.
Source: Race, statistics, and the persistent cognitive limitations of DALL-E
Assessment: UnresolvedSaid
Forecast
“though I agree with Geoff that eventually AI will have a big impact on drug design. Perhaps with current techniques, perhaps not; we will have to see.”
Our reading: AI will eventually have a big impact on drug design, possibly but not necessarily with current techniques.
No deadline stated Evidence as of
Limitation: The threshold for substantial impact is an evaluator interpretation.
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?
Assessment: UnresolvedSaid
Observation
“I think it’s fair to say that generative AI works better than most people expected.”
Our reading: Generative AI works better than most people expected.
No horizon Evidence as of
Limitation: A narrower reading about informed AI researchers could produce a different assessment.
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?
Assessment: MixedSaid
Causal claim
“It makes current AI hard to debug (nobody knows for example how to ground them in facts), and it makes them difficult predict, which means, unlike calculators or spreadsheets, we don’t really know what’s going to happen when we ask them a question.”
Our reading: Because we don't really understand how trained neural networks do what they do, current AI is hard to debug (for example, nobody knows how to ground them in facts) and difficult to predict, so unlike calculators or spreadsheets we don't really know what will happen when we ask them a question.
No horizon Evidence as of
Limitation: The supplied Anthropic passage describes interpretability obstacles, rather than directly measuring difficulty diagnosing or fixing failures; its demonstrated progress concerns a small model [e1].
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?
Assessment: SupportedSaid
Observation
“Just like we can’t guarantee that they won’t make stuff up, we don’t know how to guarantee that they won’t write flawed code.”
Our reading: We cannot guarantee that AI systems will not fabricate or write flawed code.
No horizon Evidence as of
Limitation: This is an empirical assessment of contemporary general-purpose systems, not a mathematical impossibility claim.
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?
Assessment: UnresolvedSaid
Forecast
“And emphasize that wholesale, automated fake news will be used both to manipulate markets and elections, and might undermine democracy.”
Our reading: Wholesale, automated fake news will be used to manipulate both markets and elections, and it might undermine democracy.
No deadline stated Evidence as of
Limitation: Wholesale and automated require interpretation.
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?
Assessment: UnresolvedSaid
Observation
“Your best example of a spectacular machine invention is in fact a perfect example of the kind of parroting and theft of intellectual property that is characteristic of large language models.”
Our reading: Bard's 'baby shoes' story was parroting of an existing story, and such parroting and IP theft is characteristic of large language models.
No horizon Evidence as of
Limitation: The available transcript is an updated version, and the independent transcription contains only part of the output.
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?
Assessment: SupportedSaid
Observation
“When it comes to AI in particular, we are getting way ahead of our skis, rushing forward a technology we don’t fully understand.”
Our reading: Humanity is rushing forward with AI, a technology it does not fully understand.
No horizon Evidence as of
Limitation: Support applies to deployment proceeding despite incomplete understanding.
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?
Assessment: SupportedSaid
Observation
“There’s some information in there, but whatever’s there is often both patchy and problematic.”
Our reading: Chatbots contain some information, but whatever is there is often both patchy and problematic.
No horizon Evidence as of
Limitation: Biography and family-relationship tasks are not representative of every chatbot use.
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?
Assessment: SupportedSaid
Observation
“The connections in chatbots are very different from the connections in the brain; it’s a mistake to compare them directly in this way.”
Our reading: The connections in chatbots are very different from those in the brain, so comparing them directly in the way Hinton did (by counting connections) is a mistake.
No horizon Evidence as of
Limitation: The evidence does not establish which system learns better on a specified task.
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?
Assessment: SupportedSaid
Observation
“Actually … a lot of us still think you are declaring victory prematurely.”
Our reading: A lot of people, Marcus included, still think Hinton is declaring victory for neural networks prematurely.
No horizon Evidence as of
Limitation: The result depends on reading 'premature victory' as unresolved scientific problems rather than an explicit response to Hinton.
Source: What was 60 Minutes thinking, in that interview with Geoff Hinton?