Claims checked against evidence by Gary Marcus, page 8 of 15
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Claims checked against evidence by Gary Marcus (1403)
Each of these claims was checked against dated evidence.
Page 8 of 15: claims 701 to 800 of 1403, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: SupportedSaid
Observation
“OpenAI’s problems with text and copyright started to become especially clear a year ago, when the New York Times sued them, and showed how close some of the outputs could be to some of the stories on their website.”
Our reading: OpenAI's problems with text and copyright started to become especially clear a year ago, when the New York Times sued OpenAI and showed how close some of the outputs could be to some of the stories on its website.
No horizon Evidence as of
Limitation: A complaint establishes allegations and presented evidence, not a judgment of infringement.
Source: Generative AI’s Continuing Copyright Problems, an Essay in Memory of Suchir Balaji, 1998 - 2024
Assessment: MixedSaid
Observation
“In January, in IEEE Spectrum, Reid Southen and I demonstrated similar issues in video generation software, such as Midjourney and Dall-E:”
Our reading: In January, in IEEE Spectrum, Marcus and Reid Southen demonstrated issues similar to the New York Times findings in what he called video generation software, such as Midjourney and Dall-E.
No horizon Evidence as of
Limitation: The mixed result turns on retaining the explicit word 'video'.
Source: Generative AI’s Continuing Copyright Problems, an Essay in Memory of Suchir Balaji, 1998 - 2024
Assessment: UnresolvedSaid
Observation
“OpenAI has yet to make a profit, and doesn’t anticipate making one before 2029.”
Our reading: OpenAI has not yet made a profit and does not expect to before 2029.
No horizon Evidence as of
Limitation: The original financial reporting could not be read with the required tool.
Source: Casey (Newton) Strikes Out
Assessment: UnresolvedSaid
Observation
“Newton is also conveniently ignoring an absolutely fundamental distinction, between early period of reliable, rapid progress from 2019-2022 (when training for GPT-4) was completed, and a more recent regime in which by many measures, progress has (reportedly) slowed.”
Our reading: After rapid progress in 2019-2022, LLM progress has by many measures reportedly slowed.
No horizon Evidence as of
Limitation: A narrower assessment asking only whether slowdown reports existed would be supported.
Source: Casey (Newton) Strikes Out
Assessment: UnresolvedSaid
Observation
“Diminishing returns doesn’t mean zero improvement, it means that the “laws” of scaling no longer hold, and that the ROI on adding more data and compute are no longer are predictive in the way that they used to be, hence the need for new approaches.”
Our reading: The diminishing returns he describes in AI do not mean zero improvement; they mean the scaling "laws" no longer hold and the returns on adding more data and compute are no longer predictive in the way they used to be, which is why new approaches are needed.
No horizon Evidence as of
Limitation: Important frontier training results were private.
Source: Casey (Newton) Strikes Out
Assessment: SupportedSaid
Observation
“In 2022 I warned that there would be chatbot related deaths such as suicides. This began to happen in 2023 and continued in 2024.”
Our reading: Chatbot-related deaths such as suicides occurred in 2023 and 2024.
No horizon Evidence as of
Limitation: Causation is not established. The Belgian link rests on the widow's account and chat logs she supplied. The Setzer link rests on lawsuit allegations.
Source: Casey (Newton) Strikes Out
Assessment: SupportedSaid
Observation
“Part of the value in skepticism is correctly anticipating road blocks, and suggesting alternative directions (as I have for decades with neurosymbolic AI, which is at last coming into vogue, as noted in Fortune earlier this week).”
Our reading: Neurosymbolic AI is finally coming into vogue.
No horizon Evidence as of
Limitation: The result depends on reading vogue as growing attention.
Source: Casey (Newton) Strikes Out
Assessment: UnresolvedSaid
Observation
“We still don’t have driverless cars in 97% of the world after a decade’s promises; hallucinations haven’t gone away. Home humanoid robots are nowhere in sight.”
Our reading: Driverless cars are absent from 97% of the world, hallucinations persist, and home humanoid robots are nowhere in sight.
No horizon Evidence as of
Limitation: The percentage is not reproducible without a denominator.
Source: Casey (Newton) Strikes Out
Assessment: UnresolvedSaid
Forecast
“In February, the day after Sora came out, I warned that it didn’t really understand physics—and wouldn’t — in essay called Sora’s Surreal Physics:”
Our reading: Sora does not really understand physics and will not come to understand it.
No deadline stated Evidence as of
Limitation: Understanding has no speaker-specified test or threshold.
Assessment: SupportedSaid
Observation
“That gross violation of spatiotemporal continuity/failure of object permanence is not a one-off, either; it’s something general.”
Our reading: Sora's failures of object permanence and spatiotemporal continuity are general rather than isolated.
No horizon Evidence as of
Limitation: Support rests on contemporaneous developer documentation rather than a representative independent audit.
Assessment: UnresolvedSaid
Forecast
“Sora is never really getting it, even with mountains and mountains of data.”
Our reading: Sora will never really grasp object permanence, even with vast amounts of data.
No deadline stated Evidence as of
Limitation: The cited aggregate sentence does not give a Sora-only success rate.
Assessment: UnresolvedSaid
Observation
“The third illustrates that it cannot reliably eliminate an object part.”
Our reading: ChatGPT with DALL-E 3 cannot reliably eliminate a part of an object from an image.
No horizon Evidence as of
Limitation: The packet contains no independently verified November 2024 omission outputs or directly relevant contemporary evaluation.
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: ContradictedSaid
Observation
“The remaining thirteen it got wrong in all three attempts.”
Our reading: In the authors' 17 part-specification experiments, ChatGPT with DALL-E3 got 13 wrong in all three attempts.
No horizon Evidence as of
Limitation: This is a counting finding, not an independent visual assessment of every output.
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: UnresolvedSaid
Observation
“But current technology has proven quite poor at such sanity checks. New technology may be needed.”
Our reading: Current AI technology is quite poor at checking whether its own image output satisfies the prompt and feedback, and new technology may be needed.
No horizon Evidence as of
Limitation: The evidence concerns related checking tasks, not an exact replication of the claimed workflow.
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: UnresolvedSaid
Observation
“The first example illustrates that ChatGPT with DALL-E 3 is unable to reliably associate labels correctly with the parts of an object.”
Our reading: ChatGPT with DALL-E 3 cannot reliably associate labels correctly with the parts of an object.
No horizon Evidence as of
Limitation: Unresolved does not mean the bicycle example was correct.
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: UnresolvedSaid
Observation
“In particular, DALL-E2 was very poor at following specifications about parts of objects.”
Our reading: As of October 2022, DALL-E2 was very poor at following prompt specifications about parts of objects.
No horizon Evidence as of
Limitation: This is an evidence gap, not evidence that DALL-E2 performed well.
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: UnresolvedSaid
Observation
“We did not see anything like the kind of improvement that ChatGPT has exhibited over math word problems, over the last two years.”
Our reading: In the authors' body-parts image experiments, they did not see anything like the kind of improvement that ChatGPT has shown on math word problems over the last two years.
No horizon Evidence as of
Limitation: Related benchmark gains cannot substitute for the complete comparison.
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: UnresolvedSaid
Observation
“The second example illustrates that it cannot reliably produce a non-standard number of body parts.”
Our reading: ChatGPT with DALL-E 3 cannot reliably produce images with a non-standard number of body parts.
No horizon Evidence as of
Limitation: No verified contemporary prompt-and-image evaluation directly measures requested nonstandard anatomical counts.
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: UnresolvedSaid
Observation
“This just in, right on the heels of Satya Nadella also publicly acknowledging diminishing returns:”
Our reading: Satya Nadella has publicly acknowledged diminishing returns.
No horizon Evidence as of
Limitation: Acknowledging possible limits could support a weaker interpretation of Marcus's wording.
Assessment: UnresolvedSaid
Observation
“Two years on, on the most important question of all – factuality and reliability — we are still pretty much where we were when ChatGPT first came out: wishing and hoping.”
Our reading: Two years after ChatGPT came out, on factuality and reliability, which he calls the most important question, we are still pretty much where we were at its release: wishing and hoping.
No horizon Evidence as of
Limitation: Neither study establishes aggregate progress from November 2022 to November 2024 across representative factuality and reliability tasks.
Source: ChatGPT, at Age Two
Assessment: UnresolvedSaid
Observation
“There has been a ton of experimentation, but relatively few well-documented success stories, and some fairly negative stories like a recent one at Business Insider on GPT-powered CoPilot,”
Our reading: There has been much experimentation with ChatGPT-style AI but relatively few well-documented success stories.
No horizon Evidence as of
Limitation: The survey measures deployment and measurement practices, not the number of public success stories.
Source: ChatGPT, at Age Two
Assessment: UnresolvedSaid
Observation
“But a lot of of theories about how ChatGPT would be used in practice have fizzled out.”
Our reading: Many theories about how ChatGPT would be used in practice have fizzled out.
No horizon Evidence as of
Limitation: The population of predictions is undefined.
Source: ChatGPT, at Age Two
Assessment: SupportedSaid
Observation
“Two years later Google’s search share hasn’t diminished all that much.”
Our reading: Two years after ChatGPT's launch, Google's search share has not diminished much.
No horizon Evidence as of
Limitation: The figures come from secondary reporting of Statcounter because historical primary data were not retrievable.
Source: ChatGPT, at Age Two
Assessment: UnresolvedSaid
Observation
“Yet hundreds of billions more have been invested on further speculation that scaling would somehow magically cure problems that actually appear to be inherent with the technology.”
Our reading: Hundreds of billions more have been invested on further speculation that scaling would somehow magically cure problems that appear to be inherent to the technology.
No horizon Evidence as of
Limitation: The spending categories cannot be added without checking overlap and accounting treatment.
Source: ChatGPT, at Age Two
Assessment: SupportedSaid
Observation
“GPT-4 was better than GPT-3 in almost every way; o1 is a far less uniform improvement.”
Our reading: GPT-4 was better than GPT-3 in almost every way, whereas o1 is a far less uniform improvement.
No horizon Evidence as of
Limitation: The result depends on reading the comparison qualitatively across capability categories.
Assessment: UnresolvedSaid
Observation
“But things have been changing rapidly, and the great scaling retrenchment has already begun.”
Our reading: As of November 2024, a major retreat from scaling claims in the AI field has already begun.
No horizon Evidence as of
Limitation: Great retrenchment has no stated threshold.
Assessment: SupportedSaid
Observation
“(In some cases o1 doesn’t even do better than GPT-4).”
Our reading: In some cases o1 does not perform better than GPT-4.
No horizon Evidence as of
Limitation: The supported comparison is principally against GPT-4o, not specifically the original GPT-4.
Assessment: UnresolvedSaid
Observation
“The new version is that there is not one scaling law, but three: scaling with how long you train a model (which isn’t really holding anymore), scaling with how long you post-train a model, and scaling with how long you let a given model wrestle with a given problem (or what Satya Nadella called scaling with “inference time compute”).”
Our reading: Scaling performance by training models longer is no longer really holding.
No horizon Evidence as of
Limitation: The strongest experimental counterevidence concerns an earlier model and is context only.
Assessment: UnresolvedSaid
Observation
“Even if you look at OpenAI’s own research post on o1, which introduced the newest putative law, it is immediately apparent that scaling test time compute works for some problems and not others.”
Our reading: OpenAI's own o1 research shows that scaling test-time compute helps on some problems but not others.
No horizon Evidence as of
Limitation: The conclusion depends on distinguishing model comparisons from compute-scaling experiments.
Assessment: UnresolvedSaid
Forecast
“Like the first generalization, the new ones will eventually run out.”
Our reading: The new scaling laws (post-training and inference-time) will eventually stop holding.
No deadline stated Evidence as of
Limitation: A narrower reading limited to particular scaling recipes would be easier to support.
Assessment: SupportedSaid
Observation
“Many other Tech CEOs and others, including Marc Andreeseen (A16Z), Alexandr Wang (ScaleAI), have also used the phrase “hitting a wall” and the idea of diminishing returns in the last couple weeks.”
Our reading: In the last couple of weeks before November 23, 2024, many other tech CEOs and others, including Marc Andreessen and Alexandr Wang, used the phrase "hitting a wall" and the idea of diminishing returns.
No horizon Evidence as of
Limitation: Andreessen did not use the exact phrase 'hitting a wall'; the claim is scored on the group reading.
Source: Satya Nadella and the three stages of scientific truth
Assessment: UnresolvedSaid
Observation
“Hundreds of billions of dollars were invested on the assumption that I was wrong, largely to the exclusion of investment in other approaches.”
Our reading: Hundreds of billions of dollars went into LLM scaling on the assumption that scaling would keep working, and alternative approaches got little investment.
No horizon Evidence as of
Limitation: The capex figure is a projection made in November 2024 for the full year, not a realized total.
Source: Satya Nadella and the three stages of scientific truth
Assessment: SupportedSaid
Observation
“Scaling was always a hypothesis.”
Our reading: Scaling (improving AI by adding more data and compute) was always a hypothesis.
No horizon Evidence as of
Limitation: Hypothesis does not mean unsupported speculation.
Assessment: UnresolvedSaid
Observation
“Crucially, there has never been any principled argument that they would solve hallucinations or reasoning, or edge cases in open-ended worlds —or that synthetic data would suffice indefinitely to keep feeding the machine.”
Our reading: There has never been a principled argument that scaling would solve hallucinations, reasoning, or open-world edge cases, or that synthetic data would suffice indefinitely.
No horizon Evidence as of
Limitation: The claim concerns the existence of arguments, not whether those arguments ultimately proved correct.
Assessment: SupportedSaid
Observation
“They are merely empirical generalizations that held for a certain period time, when there was enough fresh data and compute.”
Our reading: Scaling laws are merely empirical generalizations that held for a limited period while enough fresh data and compute were available.
No horizon Evidence as of
Limitation: The result depends on reading the sentence in light of the surrounding hedges about scaling having stopped.
Assessment: UnresolvedSaid
Observation
“Meanwhile, precious little investment has been made in other approaches.”
Our reading: Very little investment has gone into AI approaches other than LLMs.
No horizon Evidence as of
Limitation: The observation period is an explicit annual-reporting interpretation.
Source: CONFIRMED: LLMs have indeed reached a point of diminishing returns
Assessment: UnresolvedSaid
Forecast
“Given the cost of chips, profits will be elusive.”
Our reading: Because chips are costly, LLM companies will struggle to make profits.
No deadline stated Evidence as of
Limitation: The cited financial evidence concerns one developer.
Source: CONFIRMED: LLMs have indeed reached a point of diminishing returns
Assessment: UnresolvedSaid
Observation
“Just as I argued here in April 2024, LLMs have reached a point of diminishing returns.”
Our reading: LLMs have reached a point of diminishing returns from scaling.
No horizon Evidence as of
Limitation: The evidence lacks a comparable series of frontier training runs measuring capability gains against proportional increases in compute and data [e1, e2].
Source: CONFIRMED: LLMs have indeed reached a point of diminishing returns
Assessment: UnresolvedSaid
Forecast
“LLMs such as they are, will become a commodity; price wars will keep revenue low.”
Our reading: LLMs such as they are will become a commodity, and price wars will keep revenue low.
No deadline stated Evidence as of
Limitation: The market study is a preprint.
Source: CONFIRMED: LLMs have indeed reached a point of diminishing returns
Assessment: UnresolvedSaid
Conditional forecast
“If enthusiasm for GenAI dwindles and market valuations plummet, AI won’t disappear, and LLMs won’t disappear; they will still have their place as tools for statistical approximation.”
Our reading: If enthusiasm for GenAI dwindles and market valuations plummet, AI and LLMs will not disappear, and LLMs will still have their place as tools for statistical approximation.
No deadline stated Evidence as of
Only if: if enthusiasm for GenAI dwindles and market valuations plummet
Source: CONFIRMED: LLMs have indeed reached a point of diminishing returns
Assessment: SupportedSaid
Observation
“Many of us, including Melanie Mitchell, Subbarao Kambahapati, Emily Bender, Ernie Davis, etc. have been emphasizing for ages that there are principled limits with LLMs.”
Our reading: Marcus and many others, including Melanie Mitchell, Subbarao Kambhampati, Emily Bender and Ernie Davis, have long emphasized that there are principled limits with LLMs.
No horizon Evidence as of
Limitation: 'For ages' is colloquial, and the evidence does not establish an identical duration for every researcher.
Source: CONFIRMED: LLMs have indeed reached a point of diminishing returns
Assessment: SupportedSaid
Observation
“People aren’t ignoring GenAI; they are waiting to see if it will work.”
Our reading: People are not ignoring generative AI; they are waiting to see whether it will work.
No horizon Evidence as of
Limitation: 'People' is undefined; the strongest evidence covers businesses.
Source: AI influencer hype gone wild
Assessment: UnresolvedSaid
Observation
“He then went on to extrapolate that it would be 10,000 times better in 4 years, never considering potential bottlenecks around data, or compute, nor inherent problems with LLMs.”
Our reading: When Musk extrapolated that AI would be 10,000 times better in 4 years, he did not consider potential bottlenecks around data or compute, or inherent problems with LLMs.
No horizon Evidence as of
Limitation: The only item is one news report. It covers parts of Musk's remarks, not the full session.
Assessment: UnresolvedSaid
Causal claim
“(Side bonus: GenAI uses a massive amount of power, to the point of keeping fossil fuels in vogue, and that’s good for oil-rich countries, even if GenAI never becomes reliable.)”
Our reading: Generative AI uses so much power that it keeps fossil fuels in demand, which benefits oil-rich countries whether or not GenAI ever becomes reliable.
No horizon Evidence as of
Limitation: The evidence covers data-centre demand in general, not generative AI specifically.
Assessment: SupportedSaid
Observation
“(is OpenAI’s o1 100 times better than GPT4? Certainly not by their own data)”
Our reading: By OpenAI's own data, o1 is certainly not 100 times better than GPT-4.
No horizon Evidence as of
Limitation: OpenAI's baseline is GPT-4o, not the original GPT-4.
Assessment: UnresolvedSaid
Forecast
“My offer to bet him a million dollars on this nonsense stands.”
Our reading: AI will not be able to do anything any human can do within the next one to two years, and Marcus would bet a million dollars on it.
Not yet due Evidence as of
Limitation: The deadline falls after the evaluation date.
Assessment: UnresolvedSaid
Forecast
“It’s just fantasy to imagine selling 200 times as many humanoid robots in the nearish term when nobody knows how to build a single safe, reliable, generally useful humanoid right now, at any price.””
Our reading: Humanoid robots will not sell 200 times Roomba's roughly 50 million units in the near term.
No deadline stated Evidence as of
Limitation: The threshold inherits the approximation in Marcus's Roomba baseline.
Assessment: UnresolvedSaid
Forecast
“The same will be true for humanoid robots, and we aren’t going to see six humanoid robots for every car anytime soon.”
Our reading: Many people will not be able to afford or want humanoid robots, and there will not be six humanoid robots per car anytime soon.
No deadline stated Evidence as of
Limitation: Willingness to pay is different from inability to afford a purchase.
Assessment: UnresolvedSaid
Observation
“Meanwhile GPT-5 still hasn’t arrived, and nor has the much ballyhooed Sora.”
Our reading: As of October 2024, neither GPT-5 nor Sora has been released.
No horizon Evidence as of
Limitation: The required event-date restriction prevents the later launch announcements from serving as decisive evidence.
Assessment: UnresolvedSaid
Observation
“Those products have not by a longshot lived up to everyone’s expectations, but they do actually ship, and customers can try those products out and decide for themselves. It’s hard to call that smoke and mirrors.”
Our reading: OpenAI's products have not by a longshot lived up to everyone's expectations, but they do actually ship and customers can try them, so it is hard to call OpenAI smoke and mirrors.
No horizon Evidence as of
Limitation: Scoring only product access would omit an explicit part of the listed proposition.
Assessment: ContradictedSaid
Observation
“The difference between GPT-3 and GPT-4 was by all accounts massive; the difference between GPT-4 and o1 is by all accounts modest.”
Our reading: By all accounts the difference between GPT-3 and GPT-4 was massive, while by all accounts the difference between GPT-4 and o1 is modest.
No horizon Evidence as of
Limitation: The later comparisons use GPT-4o and o1-preview, rather than original GPT-4 and a later o1 release [e1–e3].
Assessment: UnresolvedSaid
Observation
“But then again, their products have always been served with large side order of bullshit, like when Sam said, apparently jokingly, that “AGI has been achieved internally”. None of the robot stuff they promised in the Rubik’s cube campaign ever shipped.”
Our reading: OpenAI's products have always come with lots of hype, and none of the robotics promised in its Rubik's cube campaign ever shipped.
No horizon Evidence as of
Limitation: The distinction between research ambitions and product promises materially affects the assessment.
Assessment: SupportedSaid
Observation
“Fall off on increasingly large multiplication problems has repeatedly been observed, both in older models and newer models.”
Our reading: Both older and newer LLMs have repeatedly been observed to get worse as multiplication problems get larger.
No horizon Evidence as of
Limitation: The Deng results are a researcher's social-media post plus a news report, not a peer-reviewed paper.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: MixedSaid
Observation
“I strongly believe the current results are robust.”
Our reading: Marcus strongly believes that the current results on LLMs' reasoning failures, including the Apple study and the other work he cites, are robust.
No horizon Evidence as of
Limitation: 'I strongly believe' is a statement of belief; only the robustness of the results is graded.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Observation
“Performance is ok on small problems, but quickly tails off.”
Our reading: o1 does acceptably on small planning problems, but its performance drops quickly as problems get larger.
No horizon Evidence as of
Limitation: The model tested was o1-preview.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: SupportedSaid
Observation
“Failure to follow the rules of chess is another continuing failure of formal reasoning:”
Our reading: LLMs still fail to follow the rules of chess, which is an ongoing failure of formal reasoning.
No horizon Evidence as of
Limitation: The evidence is one puzzle benchmark with a single prompt format. Other formats or full games may give different illegal-move rates.
Source: LLMs don’t do formal reasoning - and that is a HUGE problem
Assessment: UnresolvedSaid
Observation
“But the anti-neurosymbolic tradition is limited; its most visible manifestation, LLMs, has brought fame and money, but no robust solution to solving any particular problem with great reliability.”
Our reading: The anti-neurosymbolic tradition is limited. Its most visible manifestation, LLMs, has brought fame and money but no robust solution to solving any particular problem with great reliability.
No horizon Evidence as of
Limitation: High accuracy is evidence of useful capability and should not be dismissed merely because errors remain.
Assessment: ContradictedSaid
Observation
“We have been told that AI will “solve physics” (per Sam Altman), make major breakthroughs in medicine, battery technology, etc, but so far none of that has materialized.”
Our reading: None of the promised AI advances, such as 'solving physics' (per Sam Altman) or major breakthroughs in medicine and battery technology, has materialized so far.
No horizon Evidence as of
Limitation: The contradiction depends on including basic medical research within medicine; the evidence does not establish transformative clinical benefits [e1, e2].
Assessment: UnresolvedSaid
Observation
“Almost every putative virtue of AI (aside from making a small number of people rich) has been promissory.”
Our reading: Almost every claimed benefit of AI, other than enriching a few people, has so far been only a promise.
No horizon Evidence as of
Limitation: The search identified examples and a clinical review, not a representative survey of AI promises.
Assessment: UnresolvedSaid
Observation
“Even though some of the drawbacks, which inherently are less dependent on reliably and tight causal reasoning, such as misinformation use, nonconsensual deepfake porn, ans deepfake kidnapping scams, are already here to say).”
Our reading: Some of AI's drawbacks, which are inherently less dependent on reliable and tight causal reasoning, such as misinformation, nonconsensual deepfake porn, and deepfake kidnapping scams, are already here to stay.
No horizon Evidence as of
Limitation: Reported voice impersonation is not equivalent to forensic confirmation of AI use.
Assessment: UnresolvedSaid
Observation
“Other forms of AI, that are more domain-specific and specialized might help; those aren’t actually the ones getting the massive funding – or doing the massive harm.”
Our reading: Specialized, domain-specific AI might help with climate change, but it is not the kind getting massive funding or causing massive harm.
No horizon Evidence as of
Limitation: The annual target period is an explicit interpretation of an ongoing observation.
Assessment: UnresolvedSaid
Forecast
“Altman had, much to my surprise, just echoed my longstanding position that current techniques alone would not be enough to get to AGI”
Our reading: Current AI techniques alone will not be enough to reach AGI.
No deadline stated Evidence as of
Limitation: There is no generally accepted empirical test for AGI [e1].
Assessment: SupportedSaid
Observation
“I rushed out a hot take on X, my usual cautions about how LLMs aren’t really AGI:”
Our reading: LLMs, including OpenAI's new o1, are not really AGI.
No horizon Evidence as of
Limitation: The system card tests several model checkpoints and configurations; it is not an exhaustive evaluation of all contemporary LLMs [e3].
Assessment: SupportedSaid
Observation
“Almost exactly what I predicted we would see with GPT-4, back on Christmas Day 2022.”
Our reading: Altman's description of o1 as 'still flawed, still limited, seem more impressive on first use' is almost exactly what Marcus predicted we would see with GPT-4 on Christmas Day 2022.
No horizon Evidence as of
Limitation: The agreement is thematic, not verbatim.
Assessment: UnresolvedSaid
Conditional forecast
“No such governance board ever materialized; if it does, you can bet it won’t have any teeth, unless there is a government mandate that it does.”
Our reading: The governance board Altman spoke of to The New Yorker in 2016 never materialized, and Marcus bets that if one does, it won't have any teeth unless a government mandate requires it to.
No deadline stated Evidence as of
Only if: if such a board materializes
Source: Scoop: What former employees of OpenAI are worried about
Assessment: UnresolvedSaid
Observation
“I stressed the same in my Senate testimony, but nothing thus far in existing US law addresses this need.”
Our reading: So far, nothing in existing US law addresses the need for external governance under which companies cannot make decisions of potentially enormous magnitude on their own and external advisory bodies have a strong say in weighing risks against benefits.
No horizon Evidence as of
Limitation: A federal-only assessment would narrow the speaker's wording.
Source: Scoop: What former employees of OpenAI are worried about
Assessment: SupportedSaid
Observation
“California’s pending AI bill SB-1047 was supposed to be a tiny step in that direction, and even that has gotten watered down.”
Our reading: California's SB-1047, already only a small step toward external AI governance, has been watered down.
No horizon Evidence as of
Limitation: The evidence is the author's dated amendment explanation, not a complete comparison of all bill versions.
Source: Scoop: What former employees of OpenAI are worried about
Assessment: SupportedSaid
Observation
“When Sam Altman stood next to me in the Senate, he told the Senate that he strongly favored AI regulation. But behind the scenes his team fought to water down the EU AI Act.”
Our reading: Sam Altman told the Senate, standing next to Marcus, that he strongly favored AI regulation, but behind the scenes his team fought to water down the EU AI Act.
No horizon Evidence as of
Limitation: The lobbying finding relies on reputable reporting because the available primary PDF extraction is corrupted.
Assessment: UnresolvedSaid
Observation
“Companies like OpenAI that have talked big about investing in safety research and then stinted on their promises”
Our reading: OpenAI talked big about investing in safety research and then failed to deliver on those promises.
No horizon Evidence as of
Limitation: The packet does not establish which particular promise Marcus meant.
Assessment: SupportedSaid
Observation
“It also provides important whistleblower protections, which as many ex OpenAI employees have made clear, are critical.”
Our reading: SB-1047 provides important whistleblower protections, which, as many ex-OpenAI employees have made clear, are critical.
No horizon Evidence as of
Limitation: The assessment concerns a proposed bill, not protections already in force.
Assessment: ContradictedSaid
Observation
“In the new form, SB 1047 can basically only be used only after something really bad happens, as a tool to hold companies liable. It can no longer protect us against obvious negligence that might likely lead to great harm.”
Our reading: In its new form, SB-1047 can basically only be used after something really bad happens, as a tool to hold companies liable, and it can no longer protect against obvious negligence that might likely lead to great harm.
No horizon Evidence as of
Limitation: The decisive source is the bill sponsor's explanation, rather than a clean statutory transcription.
Assessment: SupportedSaid
Observation
“Most or all of the major big tech companies joined a lobbying organization that fought SB-1047, tooth and nail, despite broad public support for the bill.”
Our reading: Most or all major big tech companies joined a lobbying group that fought SB-1047 hard despite broad public support for it.
No horizon Evidence as of
Limitation: The majority assessment uses the contract's five-company group: Alphabet/Google, Amazon, Apple, Meta, and Microsoft.
Assessment: UnresolvedSaid
Observation
“And no, GPT-5 did not drop this week as many had hoped.”
Our reading: GPT-5 was not released in the week before 2024-08-18, despite many people hoping it would be.
No horizon Evidence as of
Limitation: The target-period evidence does not fully cover August 12–18, 2024 [e1, e2].
Source: What has and has not changed in the AI since the ChatGPT revolution? [Video]
Assessment: SupportedSaid
Observation
“Lastly, asking for standards is not unique to AI; it's common across many industries to ask that companies evaluate the safety of their products according to set standards: just look at the pharmaceutical industry, or aviation, automobiles, etc.”
Our reading: Asking for standards is not unique to AI: in many industries, such as pharmaceuticals, aviation and automobiles, it is common to ask companies to evaluate the safety of their products against set standards.
No horizon Evidence as of
Limitation: The evidence establishes the three named U.S. examples, not a numerical survey of every industry or jurisdiction.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047
Assessment: SupportedSaid
Observation
“And much of what would be required is already something companies committed to voluntarily, in discussions at the White House and in Seoul.”
Our reading: Much of what SB-1047 would require is something AI companies already voluntarily committed to at the White House and in Seoul.
No horizon Evidence as of
Limitation: The assessment is a qualitative comparison, not a count of statutory clauses.
Source: An open letter to Fei-Fei Li concerning California’s proposed AI regulation, SB-1047
Assessment: SupportedSaid
Observation
“OpenAI recently put Paul Nakasone (ex NSA) on the board”
Our reading: OpenAI recently put Paul Nakasone, formerly of the NSA, on its board.
No horizon Evidence as of
Limitation: This establishes the appointment and career background, not any implication about surveillance.
Source: OpenAI: On a Path to Becoming The World’s Most Frightening Surveillance Company?
Assessment: UnresolvedSaid
Observation
“Sam Altman has acknowledged wanting to train on everyone's personal documents (Word files, email etc)”
Our reading: Sam Altman has acknowledged that he wants to train AI on everyone's personal documents, such as Word files and email.
No horizon Evidence as of
Limitation: The search cannot establish that no other qualifying statement exists.
Source: OpenAI: On a Path to Becoming The World’s Most Frightening Surveillance Company?
Assessment: UnresolvedSaid
Observation
“OpenAI just put in money in a $60M fundraise with a Webcam company and is planning hardware joint venture with them.”
Our reading: OpenAI recently invested in a $60M fundraise for a webcam company and is planning a hardware joint venture with it.
No horizon Evidence as of
Limitation: The original financial report could not be read with read_page.
Source: OpenAI: On a Path to Becoming The World’s Most Frightening Surveillance Company?
Assessment: SupportedSaid
Observation
“An awful lot of key players have left the island. Most voluntarily.”
Our reading: Many key OpenAI people have left, most of them voluntarily.
No horizon Evidence as of
Limitation: The roster is a transparent group drawn from contemporaneous reporting, not a complete personnel register.
Source: The OpenAI Plot Thickens
Assessment: UnresolvedSaid
Forecast
“LLMs are not AGI, and (on their own) never will be; scaling alone was never going to be enough.”
Our reading: LLMs on their own are not and will never be AGI.
No deadline stated Evidence as of
Limitation: The review cannot establish the absence of every possible demonstration.
Assessment: UnresolvedSaid
Forecast
“LLMs are not AGI, and (on their own) never will be; scaling alone was never going to be enough.”
Our reading: Scaling alone was never going to be enough to reach AGI.
No deadline stated Evidence as of
Limitation: Benchmark correlations do not establish whether scaling alone is sufficient for AGI [e1].
Assessment: SupportedSaid
Observation
“Nvidia dropped 6%, 20% over the last month”
Our reading: Nvidia stock fell 6% on August 5, 2024, and 20% over the previous month.
No horizon Evidence as of
Limitation: The result depends on reading the figures as rounded closing-price changes.
Assessment: SupportedSaid
Observation
“OpenAI tempered expectations for its next event, and said we wouldn’t see GPT-5 then.”
Our reading: OpenAI tempered expectations for its next event and said we would not see GPT-5 at that event.
No horizon Evidence as of
Limitation: Evidence is a reporter's direct quotation, not an accessible original company announcement.
Assessment: UnresolvedSaid
Observation
“The bubble has begun to burst.”
Our reading: As of late July 2024, the GenAI bubble has begun to burst.
No horizon Evidence as of
Limitation: A narrower claim about a technology-stock correction would be easier to support.
Assessment: UnresolvedSaid
Observation
“I have long argued that Generative AI might be a dud; I just didn’t expect it to fade so fast.”
Our reading: Generative AI is fading fast, as of July 2024, and may turn out to be a dud.
No horizon Evidence as of
Limitation: The broad wording requires an evaluator-supplied reading.
Assessment: SupportedSaid
Observation
“Another GenAI monetization scheme bites the dust.”
Our reading: Meta dropping a major GenAI project is one more GenAI monetization scheme that has failed.
No horizon Evidence as of
Limitation: Shutdown evidence comes from a reputable news report quoting Meta directly; no standalone Meta shutdown announcement was recovered.
Assessment: UnresolvedSaid
Observation
“Users have lost faith, clients have lost faith, VC’s have lost faith.”
Our reading: As of July 2024, users, clients, and venture capitalists have lost faith in generative AI.
No horizon Evidence as of
Limitation: General-public opinions cannot be treated as a representative survey of GenAI users.
Assessment: UnresolvedSaid
Observation
“PS Microsoft Bing completely flubbed, 4/4, “people going yuck towards Generative AI”.”
Our reading: Microsoft Bing completely flubbed the prompt "people going yuck towards Generative AI", 4 out of 4 results.
No horizon Evidence as of
Limitation: Only one search query was run, so an independent record could still exist that this search missed.
Source: Most people don’t think GenAI has improved their productivity
Assessment: SupportedSaid
Observation
“Consequently, any “reasoning” or “planning” in LLMs is hit or miss, viable if the particulars of a given situation are close enough to what is in the training set, but highly vulnerable if they are not.”
Our reading: Marcus says any reasoning or planning in LLMs is hit or miss: viable if a situation's particulars are close enough to the training set, but highly vulnerable if they are not.
No horizon Evidence as of
Limitation: Reasoning models released after July 2024 are not assessed.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Observation
“People are starting to realize all of the above.”
Our reading: People are starting to realize generative AI has plateaued, lacks a killer app, still hallucinates and errs, and has no moat.
No horizon Evidence as of
Limitation: The population and size of the claimed change are unspecified.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Observation
“Much as I admire the new pair of systems, they do have a serious shortcoming and one that is not unfamiliar. The systems both rely on a kind of cheat on the input: trained human coders translate all the Olympiad's input sentences into mathematical form.”
Our reading: AlphaProof and AlphaGeometry 2 rely on human coders translating Olympiad problems into formal form, a serious shortcoming.
No horizon Evidence as of
Limitation: The assessment concerns the July 2024 demonstration, not later systems.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Conditional forecast
“That said, I am certain that the impending collapse won’t lead to the absolute death of AI. There is too much at stake.”
Our reading: The impending generative AI collapse will not lead to the absolute death of AI.
No deadline stated Evidence as of
Only if: after the generative AI collapse
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: ContradictedSaid
Forecast
“I fully expect that the generative AI bubble will begin to burst within the next 12 months, for many reasons:”
Our reading: The generative AI bubble will begin to burst by July 28, 2025.
Deadline passed Evidence as of
Limitation: The result depends on interpreting burst as a broad, sustained unwinding.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Forecast
“There can be no path to AGI without neurosymbolic AI.”
Our reading: AGI cannot be achieved without neurosymbolic AI.
No deadline stated Evidence as of
Limitation: Public demonstrations may not disclose enough implementation detail to classify a system's methods.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Observation
“Nobody has a moat”
Our reading: No generative AI company has a competitive moat.
No horizon Evidence as of
Limitation: The universal claim requires unusually broad evidence to support.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Observation
“Nonetheless, GDM's new work is powerful proof of concept (pun intended!). Neural networks and symbol systems can be profitably combined, contra Hinton and LeCun.”
Our reading: Google DeepMind's work shows neural networks and symbolic systems can be profitably combined.
No horizon Evidence as of
Limitation: Technical usefulness does not imply financial profitability.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Forecast
“The neural network approach known as generative AI — especially its poster chatbot ChatGPT – has captured the world’s imagination, but will soon flame out.”
Our reading: Generative AI, especially ChatGPT, will soon flame out.
No deadline stated Evidence as of
Limitation: OpenAI's disclosure is self-reported.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: SupportedSaid
Observation
“Hallucinations and boneheaded errors of reasoning and what I like to call discomprehensions are, as far as I can tell, an inevitable byproduct of LLMs; certainly they have yet to disappear, despite many promises to the contrary.”
Our reading: Marcus says that, as far as he can tell, hallucinations, boneheaded reasoning errors and 'discomprehensions' are an inevitable byproduct of LLMs, and that they certainly have not yet disappeared despite many promises to the contrary.
No horizon Evidence as of
Limitation: Only the 'yet to disappear' part is scored.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: ContradictedSaid
Observation
“The current approach has reached a plateau”
Our reading: The current generative AI approach has reached a plateau.
No horizon Evidence as of
Limitation: Arena preferences can reflect presentation as well as correctness.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic
Assessment: UnresolvedSaid
Observation
“So for the most part, we’ve been stuck in a world in which all the major tech companies are building exactly the same thing (bigger and bigger LLMs based on more data), achieving almost exactly the same results (a passel of GPT-4 level models, hardly different from each other, all struggling with hallucinations and boneheaded errors), and hardly any investment in anything else.”
Our reading: Marcus says that, for the most part, all the major tech companies are building the same thing (bigger LLMs on more data) and getting almost the same results (GPT-4-level models that barely differ and all struggle with hallucinations and boneheaded errors), with hardly any investment in anything else.
No horizon Evidence as of
Limitation: Company evaluations are not a substitute for consistent independent comparisons.
Source: AlphaProof, AlphaGeometry, ChatGPT, and why the future of AI is neurosymbolic