Unresolved claims by Gary Marcus, page 4 of 8
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Unresolved claims by Gary Marcus (772)
The record was checked, and the evidence available by the cutoff does not settle it. Forecasts whose deadline has not arrived are listed apart, as not yet due.
Page 4 of 8: claims 301 to 400 of 772, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: UnresolvedSaid
Observation
“They never let you know that a significant number of real world coders are deeply frustrated with automated coding tools like Cursor.AI (which is in part powered by Anthropic’s Claude):”
Our reading: A significant number of real-world coders are deeply frustrated with automated coding tools such as Cursor, which is partly powered by Anthropic's Claude.
No horizon Evidence as of
Limitation: Significant number and deeply frustrated remain qualitative.
Assessment: UnresolvedSaid
Observation
“Nine years later and hundreds of startups later, my understanding is that not one radiologist has been replaced, and many parts of the world actually have a shortage of radiologists.”
Our reading: Marcus's understanding is that nine years and hundreds of startups after Hinton's 2016 prediction, not one radiologist has been replaced, and many parts of the world have a shortage of radiologists.
No horizon Evidence as of
Limitation: The outcome depends on reading replacement as job displacement rather than substitution on individual tasks.
Assessment: UnresolvedSaid
Forecast
“People are increasingly going to discourage American children from learning to code based on naive enthusiasm from people like Roose.”
Our reading: Hype about AI coding will increasingly lead people to discourage American children from learning to code.
No deadline stated Evidence as of
Limitation: Public attitudes toward coding are not the same as actual advice to children.
Assessment: UnresolvedSaid
Observation
“it’s often easy to get a solution that is 80% correct, and extremely difficult to get the last 20%. Driverless cars have been like that for decades.”
Our reading: In AI it is often easy to reach 80% correctness and extremely difficult to get the last 20%, as with driverless cars for decades.
No horizon Evidence as of
Limitation: Distance automated is not equivalent to correctness across all driving situations.
Assessment: UnresolvedSaid
Observation
“Even Altman himself was uncharacteristically tepid in his product announcement.”
Our reading: Sam Altman was unusually tepid in announcing GPT-4.5, not mentioning AGI.
No horizon Evidence as of
Limitation: The packet provides only one earlier product announcement, which cannot establish Altman's usual tone. [e3]
Assessment: UnresolvedSaid
Observation
“Gary Marcus was right all along that pure scaling would eventually hit a wall, or at least reach a point of massively diminishing returns.”
Our reading: Pure scaling of LLMs has hit a wall or at least massively diminishing returns.
No horizon Evidence as of
Limitation: The launch comparisons do not isolate the contribution of pretraining scale or provide a controlled measure of returns to additional resources. [e1]
Assessment: UnresolvedSaid
Observation
“Right now, the math for profitability doesn’t work, influence on productivity of GenAI has been minimal, and prospects for pure scaling aren’t looking great.”
Our reading: The math for generative AI profitability currently doesn't work.
No horizon Evidence as of
Limitation: Available loss reporting is not an industry financial statement.
Source: GPT 4.5 is no GPT-5
Assessment: UnresolvedSaid
Observation
“Right now, the math for profitability doesn’t work, influence on productivity of GenAI has been minimal, and prospects for pure scaling aren’t looking great.”
Our reading: Generative AI's influence on productivity has been minimal.
No horizon Evidence as of
Limitation: The original survey estimates were subsequently revised.
Source: GPT 4.5 is no GPT-5
Assessment: UnresolvedSaid
Observation
“It’s not just OpenAI either; it’s the whole field, as I discussed in my own recent Fortune article:”
Our reading: Diminishing returns from scaling affect the whole AI field, not just OpenAI.
No horizon Evidence as of
Limitation: Important underlying experiments were confidential.
Source: GPT 4.5 is no GPT-5
Assessment: UnresolvedSaid
Observation
“The truth is that scaling just isn’t producing the fruit that it once did.”
Our reading: Scaling is no longer producing the gains it once did.
No horizon Evidence as of
Limitation: The available reporting is more specific to particular projects than the claim.
Source: GPT 4.5 is no GPT-5
Assessment: UnresolvedSaid
Observation
“And for that matter people should be comparing GenAI revenue (on the order of low tens of billions per year, pre price wars) with GenAI expenditures (order half a trillion dollars to date), and doing the math, especially given the price wars induced by DeepSeek.”
Our reading: GenAI revenue is on the order of low tens of billions of dollars per year (before the price wars), while GenAI expenditures are on the order of half a trillion dollars to date.
No horizon Evidence as of
Limitation: The claim combines different periods and incompletely defined industry boundaries.
Source: GPT 4.5 is no GPT-5
Assessment: UnresolvedSaid
Causal claim
“I personally noticed Musk’s shutting out of opposing views when he stopped answering my emails when I warned him in July – correctly - that throwing in with Trump would undermine the cause of AI safety (which for a decade, until a few weeks ago, seemed extremely important to Musk).”
Our reading: Musk's alignment with Trump has undermined the cause of AI safety, as Marcus correctly warned in July 2024.
No horizon Evidence as of
Limitation: No admitted evidence establishes Musk's causal contribution to the federal advanced-AI policy rollback.
Source: Elon Musk's inability to listen to others is torching almost everything he touches
Assessment: UnresolvedSaid
Observation
“Taking this into account, the fraction of problems where a composite AI system could solve the problem starting with its English language statement is at most 60%.”
Our reading: Of the 50 IMO geometry problems from 2000 to 2024, a composite AI system starting from the English problem statements could solve at most 60%.
No horizon Evidence as of
Limitation: Automatic translation success does not itself establish solving success.
Source: AlphaGeometry2: Impressive accomplishment, but still a long path ahead
Assessment: UnresolvedSaid
Observation
“DeepMind didn’t make that claim, and it isn’t true.”
Our reading: The TechCrunch headline's claim about AlphaGeometry2 was not made by DeepMind and is not true.
No horizon Evidence as of
Limitation: The packet contains no quoted DeepMind report or matched performance evidence.
Source: AlphaGeometry2: Impressive accomplishment, but still a long path ahead
Assessment: UnresolvedSaid
Conditional forecast
“Even if one were to glom together the best general AIs with AlphaGeometry2, the combination would fall far short of the geometric understanding and problem solving ability of high school math student with a talent for math.”
Our reading: Combining the best general AIs with AlphaGeometry2 would still fall far short of a talented high school math student's geometric understanding and problem-solving.
No deadline stated Evidence as of
Only if: if the best general AIs were combined with AlphaGeometry2
Source: AlphaGeometry2: Impressive accomplishment, but still a long path ahead
Assessment: UnresolvedSaid
Observation
“Over the last couple days I have been asking version 2 to draw images identifying and tagging parts of images. The results are consistently awful:”
Our reading: When Marcus asked Grok 2 over the last couple of days to draw images identifying and tagging parts of images, the results were consistently awful.
No horizon Evidence as of
Limitation: Text extraction cannot establish what the embedded images show.
Assessment: UnresolvedSaid
Forecast
“And it will be the citizens of the United States who bear the consequences, while the guy who stood to get a $400M armored T̶e̶s̶l̶a̶ EV government contract (until the media squawked) stands to collect a big slice of the profits, unemploying thousands of lifelong government employees.”
Our reading: US citizens will bear the consequences, while Musk stands to collect a big slice of the profits, putting thousands of lifelong government employees out of work.
No deadline stated Evidence as of
Limitation: Employment totals combine different reasons for departure and do not establish AI causation.
Assessment: UnresolvedSaid
Observation
“Yet Musk wants to use shove problematic AI, both biased and unreliable, down our throats, immediately, as reported by The New York Times, and noted in this Bluesky comment:”
Our reading: Musk wants to deploy biased and unreliable AI in government immediately.
No horizon Evidence as of
Limitation: The usable evidence is news reporting based on sources describing an internal meeting; no primary meeting record was successfully read.
Assessment: UnresolvedSaid
Forecast
“Soon such unreliable stuff, with butt-covering warning messages that most people will ignore, will be everywhere.”
Our reading: Soon such unreliable AI, with butt-covering warning messages that most people will ignore, will be everywhere.
No deadline stated Evidence as of
Limitation: The search does not establish that qualifying evidence cannot exist.
Assessment: UnresolvedSaid
Forecast
“For nearly three years I have been saying that the pure scaling of LLMs – adding more data and more parameters – would eventually run out, and that it would fail to solve hallucinations and boneheaded errors and that scaling laws were merely empirical generalizations, rather than physical laws.”
Our reading: Marcus restates his long-held prediction that pure scaling of LLMs (adding more data and more parameters) would eventually run out and would fail to solve hallucinations and boneheaded errors, and that scaling laws are merely empirical generalizations rather than physical laws.
No deadline stated Evidence as of
Limitation: Available model comparisons change several factors simultaneously.
Source: Breaking: OpenAI's efforts at pure scaling have hit a wall.
Assessment: UnresolvedSaid
Observation
“They were overhyping LLMS as AGI, even as problems like hallucinations and boneheaded persisted.”
Our reading: As a premise of Taming Silicon Valley, tech oligarchs were overhyping LLMs as AGI even as problems like hallucinations and boneheaded errors persisted.
No horizon Evidence as of
Limitation: The packet lacks a specific executive AGI claim matched to contemporary evaluations of the corresponding systems and capabilities.
Source: Everything I warned about in Taming Silicon Valley is rapidly becoming our reality
Assessment: UnresolvedSaid
Causal claim
“(Also, having our own massive LLMs will do nothing to stop China from using theirs in authoritarian pursuits.)”
Our reading: US possession of massive LLMs will do nothing to stop China from using its LLMs for authoritarian purposes.
No horizon Evidence as of
Limitation: Public records provide limited visibility into covert deployments and their outcomes.
Source: Everything I warned about in Taming Silicon Valley is rapidly becoming our reality
Assessment: UnresolvedSaid
Observation
“LLMs, once novel, are largely a commodity.”
Our reading: LLMs have largely become a commodity.
No horizon Evidence as of
Limitation: The survey’s 2024 findings cannot be isolated to the target window [e1].
Assessment: UnresolvedSaid
Observation
“(Note that test-time compute systems are not across the board improvements like GPT-4 was relative to GPT-3 or GPT-3 was relative to GPT-2, but rather improvements in certain domains such as coding and math.)”
Our reading: Test-time compute systems improve only certain domains like coding and math, not across the board as GPT-4 did over GPT-3.
No horizon Evidence as of
Limitation: The decisive model evaluations are developer-reported.
Assessment: UnresolvedSaid
Forecast
“A key claim of the paper was that pure scaling of LLMs – just adding more data and compute to extant LLM architectures – would not bring us to AGI, and that so-called scaling laws were merely empirical generalizations rather than physical laws.”
Our reading: Pure scaling of LLMs by adding data and compute to existing architectures will not reach AGI; scaling laws are empirical generalizations, not physical laws.
No deadline stated Evidence as of
Limitation: Public model documentation may not disclose enough to establish scaling-only provenance.
Assessment: UnresolvedSaid
Causal claim
“What changed here was the AI landscape. It’s all about money, of course.”
Our reading: Google's shift toward military and surveillance AI was caused by financial pressure in the changed AI landscape.
No horizon Evidence as of
Limitation: Public explanations can omit commercial considerations.
Source: Google, 2001: Don’t Be Evil
Assessment: UnresolvedSaid
Forecast
“If they were already making zillions from AI, they would have mentioned that on the investor call. But presumably they are not, and that cannot continue indefinitely.”
Our reading: Google is presumably not already making zillions from AI (or it would have said so on the investor call), and that situation cannot continue indefinitely.
No deadline stated Evidence as of
Limitation: The baseline premise remains unverified.
Source: Google, 2001: Don’t Be Evil
Assessment: UnresolvedSaid
Causal claim
“Google has to find a way to make back the money back. Bye bye corporate motto, hello military and surveillance.”
Our reading: Pressure to recoup AI investment is driving Google to abandon its motto and pursue military and surveillance applications.
No horizon Evidence as of
Limitation: The packet contains no decision records or attributable testimony establishing whether pressure to recover AI investment materially influenced the change.
Source: Google, 2001: Don’t Be Evil
Assessment: UnresolvedSaid
Observation
“Reading between the lines, Google is locked in a epic arms race for AI supremacy that is not so far paying off.”
Our reading: Judging by its earnings call omissions, Google's AI arms race spending has not paid off so far.
No horizon Evidence as of
Limitation: A narrower claim that Google reported no AI revenue would face contrary evidence, but that is not the proposition being graded.
Source: Google, 2001: Don’t Be Evil
Assessment: UnresolvedSaid
Observation
“failed, multiple times, to include a full list of all US states”
Our reading: In Marcus's test, the free version of ChatGPT failed several times to produce a complete list of all 50 US states.
No horizon Evidence as of
Limitation: The shared transcript linked in Marcus's post was not found, so none of his tables could be checked against the list of 50 states.
Source: ChatGPT in Shambles
Assessment: UnresolvedSaid
Observation
“issued numerous corrections that were wrong, never acknowledging uncertainty until after its errors were called out.”
Our reading: In Marcus's test, ChatGPT issued many wrong corrections and acknowledged no uncertainty until its errors were pointed out.
No horizon Evidence as of
Limitation: No substantive conversation messages were available as admissible outcome evidence.
Source: ChatGPT in Shambles
Assessment: UnresolvedSaid
Observation
“failed, multiple times, to properly count to 50”
Our reading: In Marcus's test, ChatGPT failed several times to count to 50 correctly.
No horizon Evidence as of
Limitation: The shared transcript linked in Marcus's post was not found, so the rows in his tables could not be counted.
Source: ChatGPT in Shambles
Assessment: UnresolvedSaid
Observation
“reported that the letter h could be a vowel, at least when it appeared in the word Northwest”
Our reading: In Marcus's test, ChatGPT reported that the letter h could be a vowel, at least in the word Northwest.
No horizon Evidence as of
Limitation: The shared transcript and the screenshots of the Northwest Territories answer were not found.
Source: ChatGPT in Shambles
Assessment: UnresolvedSaid
Forecast
“We can be sure that DeepResearch will get heavily used.”
Our reading: Deep Research will be heavily used.
No deadline stated Evidence as of
Limitation: The disclosure does not quantify Deep Research's absolute scale or frequency of use.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: UnresolvedSaid
Forecast
“Deep Research, because it works faster and expands the reach of what can be automatically generated, is going to make that problem worse.”
Our reading: Deep Research is going to make the problem of the internet being polluted with AI-generated garbage content worse, because it works faster and expands the reach of what can be automatically generated.
No deadline stated Evidence as of
Limitation: Missing product attribution is not evidence that no contribution occurred.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: UnresolvedSaid
Forecast
“Unfortunately, virtually everything that Deep Research produces will pass a LGTM test ( “it ‘looks good to me’”); people will assume it’s legit, even when it is isn’t.”
Our reading: Virtually all Deep Research output will superficially look good and people will assume it is legitimate even when it is not.
No deadline stated Evidence as of
Limitation: The absence of a located study is an evidence gap, not proof that the predicted behavior does not occur.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: UnresolvedSaid
Forecast
“The output of DeepResearch will inevitably feed further models; we will wind up with more of this:”
Our reading: Deep Research output will inevitably be used to train further models, contributing to model collapse.
No deadline stated Evidence as of
Limitation: Training datasets may not disclose product-level provenance.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: UnresolvedSaid
Forecast
“Few people will fact-check its output carefully, since it will all superficially look ok.”
Our reading: Few people will carefully fact-check Deep Research's output, since it will all superficially look ok.
No deadline stated Evidence as of
Limitation: Checking behavior may vary substantially with expertise and task stakes.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: UnresolvedSaid
Forecast
“Scammers will use it, for example, to write “medical reports” that are half-baked (in order to sell ads on the webistes that report such “research”).”
Our reading: Scammers will use Deep Research to write half-baked 'medical reports' to sell ads on websites.
No deadline stated Evidence as of
Limitation: The search cannot establish that no qualifying case exists.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: UnresolvedSaid
Forecast
“Because Deep Research lowers the effort, making it trivial to write legit-sounding articles, paper mill factories (there is already such a thing, designed to game citation counts, “an illegitimate, yet lucrative business“ that is growing) will massively ramp up production, choking the peer review process and saturating the market with hard-to-find, hard-to-rectify errors.”
Our reading: Because Deep Research lowers the effort and makes it trivial to write legit-sounding articles, paper mills will massively ramp up production, choking the peer review process and saturating the market with hard-to-find, hard-to-rectify errors.
No deadline stated Evidence as of
Limitation: The search cannot establish that no undisclosed paper-mill use exists.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: UnresolvedSaid
Forecast
“And it’s not just the models that will be infected, but the scientific literature itself.”
Our reading: The scientific literature itself, not just models, will be infected by Deep Research output.
No deadline stated Evidence as of
Limitation: This assessment uses the contextual meaning of infected as erroneous contamination.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: UnresolvedSaid
Forecast
“When — not if — Deep Research begins to flood the medical zone with shite, we are all in trouble.”
Our reading: Deep Research will flood the medical field with garbage content, causing serious trouble.
No deadline stated Evidence as of
Limitation: Benchmark errors establish capability limitations, not the prevalence of harmful medical content in circulation.
Source: Deep Research, Deep Bullshit, and the potential (model) collapse of science
Assessment: UnresolvedSaid
Observation
“DeepSeek r1 is not smarter than earlier models, just trained more cheaply”
Our reading: DeepSeek r1 was trained more cheaply than earlier models.
No horizon Evidence as of
Limitation: Unresolved does not mean the cheaper-training claim is false.
Source: Five things most people don't seem to understand about DeepSeek
Assessment: UnresolvedSaid
Observation
“It is still expensive to operate ("inference"), especially if you want to make it "think" longer like o3.”
Our reading: DeepSeek r1 is still expensive to run at inference, especially when made to reason longer like o3.
No horizon Evidence as of
Limitation: The claim has a meaningful ordinary reading, but its cost level remains unmeasured.
Source: Five things most people don't seem to understand about DeepSeek
Assessment: UnresolvedSaid
Forecast
“The biggest threat isn't to Nvidia (people will still need GPUs, albeit fewer higher end ones), but rather to OpenAI and Anthropic (because price wars will undercut their hopes of making a profit).”
Our reading: The biggest threat from DeepSeek is not to Nvidia, because people will still need GPUs, albeit fewer higher-end ones.
No deadline stated Evidence as of
Limitation: Shipment mix is not a count of GPU requirements.
Source: Five things most people don't seem to understand about DeepSeek
Assessment: UnresolvedSaid
Forecast
“The biggest threat isn't to Nvidia (people will still need GPUs, albeit fewer higher end ones), but rather to OpenAI and Anthropic (because price wars will undercut their hopes of making a profit).”
Our reading: The biggest threat from DeepSeek is to OpenAI and Anthropic, because price wars will undercut their hopes of making a profit.
No deadline stated Evidence as of
Limitation: Pricing and product-margin statements are not company-wide net-profit measurements.
Source: Five things most people don't seem to understand about DeepSeek
Assessment: UnresolvedSaid
Forecast
“The biggest threat isn't to Nvidia (people will still need GPUs, albeit fewer higher end ones), but rather to OpenAI and Anthropic (because price wars will undercut their hopes of making a profit).”
Our reading: People will still need GPUs, though fewer higher-end ones, after DeepSeek.
No deadline stated Evidence as of
Limitation: Actual shipment mix and projected shipment growth measure different things.
Source: Five things most people don't seem to understand about DeepSeek
Assessment: UnresolvedSaid
Causal claim
“We accidentally upped their technical game.”
Our reading: US export controls accidentally improved China's AI technical capabilities by forcing efficiency innovations.
No horizon Evidence as of
Limitation: Technical development accounts come primarily from DeepSeek.
Source: The race for "AI Supremacy" is over — at least for now.
Assessment: UnresolvedSaid
Observation
“Meanwhile, revenue (order of $5B) has been modest, compared to outlays that have been much greater; no actual killer app (relative to the costs) has emerged.”
Our reading: OpenAI's revenue (on the order of $5B) has been modest compared with much greater outlays, and no actual killer app, relative to the costs, has emerged.
No horizon Evidence as of
Limitation: Contemporary annual figures were projections [e1, e2].
Source: The race for "AI Supremacy" is over — at least for now.
Assessment: UnresolvedSaid
Observation
“In late December, a company called DeepSeek, apparently initially built for quantitative trading rather than LLMs, produced a nearly state-of-the-art model that required only roughly 1/50th of the training costs of previous models, — instantly putting them in the big leagues with American companies like OpenAI, Google, and Anthropic, both in terms of performance and innovation.”
Our reading: In late December, DeepSeek, apparently first built for quantitative trading, produced a nearly state-of-the-art model that needed roughly 1/50th of the training costs of previous models, putting it in the big leagues with American companies such as OpenAI, Google and Anthropic in both performance and innovation.
No horizon Evidence as of
Limitation: Unknown accounting basis prevents a supported or contradicted financial verdict.
Source: The race for "AI Supremacy" is over — at least for now.
Assessment: UnresolvedSaid
Forecast
“I do genuinely think we will all have our own AI agents, and companies will have armies of them.”
Our reading: In the long term, everyone will have their own AI agents and companies will have armies of them.
No deadline stated Evidence as of
Limitation: The packet has no population-level data on personal AI agent use.
Assessment: UnresolvedSaid
Forecast
“And they will be worth trillions, since eventually (no time soon) they will do a huge fraction of all human knowledge work, and maybe physical labor too.”
Our reading: Eventually, though not soon, AI agents will do a huge fraction of human knowledge work, perhaps physical labor too, and be worth trillions.
No deadline stated Evidence as of
Limitation: The only evidence is one benchmark from late 2025. It measures what agents can do on a set of projects, not what share of real knowledge work they actually do.
Assessment: UnresolvedSaid
Observation
“No system has been shown to be robust across all the variations (though specific ones vary across replications, perhaps in response to publicly shared examples).”
Our reading: No system has been shown to be robust across all the variations of the posted LLM failure examples under discussion (such as those from Mahabal and Fraser), though which specific examples fail varies across replications, perhaps in response to publicly shared examples.
No horizon Evidence as of
Limitation: Search failure cannot establish that no qualifying demonstration existed.
Assessment: UnresolvedSaid
Observation
“But they remain superficial, in that their responses invariably have the right kind of language for any given problem, but (unlike the hypothetical Star Trek computer) rarely trustworthy.”
Our reading: LLMs remain superficial: their responses invariably have the right kind of language for any given problem, but they are rarely trustworthy.
No horizon Evidence as of
Limitation: The cited evaluations predate the target snapshot and are context only.
Assessment: UnresolvedSaid
Observation
“Some people shift their goalposts; others try to duck specifying goalposts altogether.”
Our reading: In AGI discussions, some people shift their goalposts while others, including Altman, avoid specifying any.
No horizon Evidence as of
Limitation: Omission, inconsistency, and deliberate evasion are different propositions.
Source: The Five Stages of AGI Grief
Assessment: UnresolvedSaid
Observation
“That’s just obviously not so, e.g., “write a biography of X individual without making stuff up” remains pretty unreliable; same for frontier models if you ask them to “play grandmaster level chess without making up illegal moves”.”
Our reading: It is obviously not true that frontier language models can perform competently at pretty much any information task that humans can do, that can be posed and answered in natural language, and that has quantifiable performance; for example, writing a biography of an individual without making things up remains pretty unreliable, as does playing grandmaster-level chess without making illegal moves.
No horizon Evidence as of
Limitation: The unresolved result reflects missing matched evidence, not evidence that the models were reliable.
Source: The Five Stages of AGI Grief
Assessment: UnresolvedSaid
Causal claim
“As I noted in 2001, the lack of distinct, accessible, reliable database-style records leads to hallucinations.”
Our reading: The lack of distinct, accessible, reliable database-style records leads to hallucinations, as Marcus noted in 2001.
No horizon Evidence as of
Limitation: The historical search was not an exhaustive examination of every 2001 publication.
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: UnresolvedSaid
Observation
“Even o1 is losing money; o3 isn’t even available, and perhaps cannot be run at a profit.]”
Our reading: OpenAI's o1 is losing money.
No horizon Evidence as of
Limitation: The evidence covers only the ChatGPT Pro tier. It does not cover o1 sold through other channels, such as ChatGPT Plus or the API.
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: UnresolvedSaid
Causal claim
“Any one example can be fixed, but examples keep rolling in because no robust solution has been found.”
Our reading: LLM commonsense failure examples keep appearing because no robust solution has been found, even though individual examples can be fixed.
No horizon Evidence as of
Limitation: Spatial reasoning studies cover only part of commonsense reasoning.
Assessment: UnresolvedSaid
Observation
“To date, though simulation has proved extremely valuable for all kinds of scientific and engineering applications, as well as video games and in creating CGI for films, it has proven much less effective for commonsense reasoning for AI.”
Our reading: Physical simulation has proven much less effective for AI commonsense reasoning than for scientific, engineering, and entertainment applications.
No horizon Evidence as of
Limitation: The evidence supplies no common effectiveness measure or broad comparison across the named domains.
Assessment: UnresolvedSaid
Observation
“Three approaches to commonsense reasoning have become popular in the last fifteen years; none has proved fully satisfactory.”
Our reading: None of the three popular approaches to commonsense reasoning (simulation, LLM emergence, video generation) has proved fully satisfactory.
No horizon Evidence as of
Limitation: Partial support for two approaches is insufficient to score the entire conjunction supported.
Assessment: UnresolvedSaid
Observation
“Two and a half years later, more recent models no longer makes the same specific mistakes that GPT-3 did, but still lack a robust understanding of the real external world.”
Our reading: Recent LLMs no longer make GPT-3's specific mistakes but still lack a robust understanding of the real external world.
No horizon Evidence as of
Limitation: The packet contains no independent matched retests of the cited GPT-3 failure cases.
Assessment: UnresolvedSaid
Causal claim
“For example, OpenAI’s o1 is able to create and execute Python code that can do geometric calculation, but because it does not understand how objects are embedded in space and interact in space or, indeed, what spatial relationships actually mean, it often makes very basic mistakes.”
Our reading: OpenAI's o1 often makes very basic spatial mistakes because it does not understand how objects are embedded and interact in space.
No horizon Evidence as of
Limitation: The cited studies do not establish the behavior of released o1 during the target period.
Assessment: UnresolvedSaid
Causal claim
“Rosie the Robot is not doing our housework, and that is in part because the kind of physical reasoning a household robot would need to do can’t be done purely by running simulations.”
Our reading: General-purpose household robots like Rosie the Robot are not doing our housework, and this is in part because the physical reasoning such a robot would need cannot be done purely by running simulations.
No horizon Evidence as of
Limitation: The available experiments cover selected manipulation tasks.
Assessment: UnresolvedSaid
Observation
“My seven December 2022 predictions about GPT-4 largely held true (number seven still pending) for every model that was released in 2024.”
Our reading: His seven December 2022 predictions about GPT-4 largely held true for every model released in 2024, with prediction number seven still pending.
No horizon Evidence as of
Limitation: Selected benchmarks cannot establish a universal claim.
Assessment: UnresolvedSaid
Forecast
“Few if any radiologists will be replaced by AI (contra Hinton’s infamous 2016 prediction).”
Our reading: By end of 2025 few if any radiologists will be replaced by AI.
Deadline passed Evidence as of
Limitation: The strongest retrieved workforce evidence concerns the UK.
Assessment: UnresolvedSaid
Forecast
“Less than 10% of the work force will be replaced by AI. Probably less than 5%.”
Our reading: By end of 2025 less than 10%, probably less than 5%, of the workforce will be replaced by AI.
Deadline passed Evidence as of
Limitation: Exposure, announced cuts and firm employment effects do not measure realized gross worker replacement [e1, e2, e3].
Assessment: UnresolvedSaid
Observation
“AI-generated misinformation became increasingly widespread, as I warned in December 2022, but the effects have – thus far — been less than I feared. I continue to be quite concerned, and think vigilance continues to be required.”
Our reading: AI-generated misinformation became increasingly widespread, but its effects have thus far been less than Marcus feared; he remains quite concerned and thinks vigilance continues to be required.
No horizon Evidence as of
Limitation: Site discovery counts do not establish audience exposure, the proportion of content that is false, or how much growth reflects monitoring coverage [e6, e7].
Assessment: UnresolvedSaid
Observation
“The company has not returned any profits, and is not expecting to do so until at least 2029, but has a burn rate in the billions, and still lacks a clear technical moat.”
Our reading: OpenAI has no profits, does not expect any until at least 2029, burns billions, and lacks a clear technical moat.
No horizon Evidence as of
Limitation: The financial evidence concerns estimates and expectations, not audited realized results.
Assessment: UnresolvedSaid
Observation
“Corporate adoption is far more limited than most people expected, and total profits across all companies (except of course hardware companies like NVidia, which profits from chips rather than models) have been modest at best.”
Our reading: Corporate adoption of AI is far more limited than most people expected, and total profits across all companies, except hardware companies like Nvidia that profit from chips rather than models, have been modest at best.
No horizon Evidence as of
Limitation: US broad-AI adoption is only context for corporate generative-AI adoption.
Assessment: UnresolvedSaid
Forecast
“Truly driverless cars, in which no human is required to attend to traffic, will continue to see usage limited to a modest number of cities, mainly in the West, mainly in good weather. Human drivers will still make up a large part of the economy.”
Our reading: By the end of 2025 truly driverless cars, in which no human is required to attend to traffic, will continue to see usage limited to a modest number of cities, mainly in the West and mainly in good weather, and human drivers will still make up a large part of the economy.
Deadline passed Evidence as of
Limitation: Operator announcements are not a global deployment census.
Assessment: UnresolvedSaid
Observation
“My predictions about the technical and economic limits of generative AI have largely been on target, but I was flat out wrong about investors.”
Our reading: His predictions about generative AI's technical and economic limits were largely correct, but he was wrong about investor behavior.
No horizon Evidence as of
Limitation: Only the financing component is independently established here.
Assessment: UnresolvedSaid
Observation
“Such pretraining is not uncommon in the field, but was not made clear in the video.”
Our reading: Such pretraining is not uncommon in the field, but OpenAI's video did not make clear that o3 had been pretrained on ARC examples.
No horizon Evidence as of
Limitation: Clarity requires considering slides as well as spoken words.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: UnresolvedSaid
Observation
“What we saw is not AGI.”
Our reading: o3 as demonstrated is not AGI.
No horizon Evidence as of
Limitation: This assesses the categorical wording, not the narrower claim that ARC performance alone did not prove AGI.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: UnresolvedSaid
Causal claim
“Same was true of the openAI graph: the MIT work (halfway in between o1 and o3) and many others results weren’t shown, making the breakthrough relative to the field seem far bigger than it really was.”
Our reading: OpenAI's graph left out the MIT work (halfway between o1 and o3) and many other results, which made o3's breakthrough relative to the field seem far bigger than it really was.
No horizon Evidence as of
Limitation: The identified MIT figure is an ensemble result, not simply the standalone test-time-training model.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: UnresolvedSaid
Causal claim
“Because the wrong experiment was performed, and key data weren’t given, we can’t compare directly with humans. (And best humans still outperformed the model).”
Our reading: Because the wrong experiment was performed and key data were not given, o3 cannot be compared directly with humans on ARC, and the best humans still outperformed the model.
No horizon Evidence as of
Limitation: No matched evaluation of December o3 and top individual human solvers is supplied.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: UnresolvedSaid
Observation
“Two graphs, one presented by OpenAI and one by Chollet were misleading.”
Our reading: The o3 ARC graphs presented by OpenAI and by Chollet were misleading.
No horizon Evidence as of
Limitation: An image's absence from extracted text cannot establish an omission within that image.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: UnresolvedSaid
Observation
“Unfortunately the key test was not done.”
Our reading: The key test of o3's base model on ARC without pretraining was not done.
No horizon Evidence as of
Limitation: A narrower reading confined to ARC Prize's own evaluation would be easier to support.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: UnresolvedSaid
Forecast
“In 2025, a lot of AI influencers and maybe some companies are going to claim we have reached AGI.”
Our reading: During 2025, many AI influencers and possibly some companies will claim AGI has been reached.
Deadline passed Evidence as of
Limitation: The evidence is a selected set of statements, not a census.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: UnresolvedSaid
Observation
“Even setting aside the 1000x costs, not one person outside of OpenAI has evaluated o3’s robustness across different types of problems.”
Our reading: No one outside OpenAI has evaluated o3's robustness across different problem types.
No horizon Evidence as of
Limitation: A broader reading of problem types could count ARC tasks and produce a different assessment.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: UnresolvedSaid
Observation
“Most commentary has ignored one or both of these ideas.”
Our reading: Most commentary on o3 has ignored one or both of two ideas: that passing ARC may be necessary but not sufficient for AGI, and that a mechanism can be effective in some domains but not others.
No horizon Evidence as of
Limitation: The packet provides no representative sampling frame, denominator, systematic coding, or uncertainty estimate for commentary published between the December 20 announcement and Marcus’s December 21 post.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: UnresolvedSaid
Observation
“OpenAI has yet to make a profit, and doesn’t anticipate making one before 2029.”
Our reading: OpenAI has not yet made a profit and does not expect to before 2029.
No horizon Evidence as of
Limitation: The original financial reporting could not be read with the required tool.
Source: Casey (Newton) Strikes Out
Assessment: UnresolvedSaid
Observation
“Newton is also conveniently ignoring an absolutely fundamental distinction, between early period of reliable, rapid progress from 2019-2022 (when training for GPT-4) was completed, and a more recent regime in which by many measures, progress has (reportedly) slowed.”
Our reading: After rapid progress in 2019-2022, LLM progress has by many measures reportedly slowed.
No horizon Evidence as of
Limitation: A narrower assessment asking only whether slowdown reports existed would be supported.
Source: Casey (Newton) Strikes Out
Assessment: UnresolvedSaid
Observation
“Diminishing returns doesn’t mean zero improvement, it means that the “laws” of scaling no longer hold, and that the ROI on adding more data and compute are no longer are predictive in the way that they used to be, hence the need for new approaches.”
Our reading: The diminishing returns he describes in AI do not mean zero improvement; they mean the scaling "laws" no longer hold and the returns on adding more data and compute are no longer predictive in the way they used to be, which is why new approaches are needed.
No horizon Evidence as of
Limitation: Important frontier training results were private.
Source: Casey (Newton) Strikes Out
Assessment: UnresolvedSaid
Observation
“We still don’t have driverless cars in 97% of the world after a decade’s promises; hallucinations haven’t gone away. Home humanoid robots are nowhere in sight.”
Our reading: Driverless cars are absent from 97% of the world, hallucinations persist, and home humanoid robots are nowhere in sight.
No horizon Evidence as of
Limitation: The percentage is not reproducible without a denominator.
Source: Casey (Newton) Strikes Out
Assessment: UnresolvedSaid
Forecast
“In February, the day after Sora came out, I warned that it didn’t really understand physics—and wouldn’t — in essay called Sora’s Surreal Physics:”
Our reading: Sora does not really understand physics and will not come to understand it.
No deadline stated Evidence as of
Limitation: Understanding has no speaker-specified test or threshold.
Assessment: UnresolvedSaid
Forecast
“Sora is never really getting it, even with mountains and mountains of data.”
Our reading: Sora will never really grasp object permanence, even with vast amounts of data.
No deadline stated Evidence as of
Limitation: The cited aggregate sentence does not give a Sora-only success rate.
Assessment: UnresolvedSaid
Observation
“The third illustrates that it cannot reliably eliminate an object part.”
Our reading: ChatGPT with DALL-E 3 cannot reliably eliminate a part of an object from an image.
No horizon Evidence as of
Limitation: The packet contains no independently verified November 2024 omission outputs or directly relevant contemporary evaluation.
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: UnresolvedSaid
Observation
“But current technology has proven quite poor at such sanity checks. New technology may be needed.”
Our reading: Current AI technology is quite poor at checking whether its own image output satisfies the prompt and feedback, and new technology may be needed.
No horizon Evidence as of
Limitation: The evidence concerns related checking tasks, not an exact replication of the claimed workflow.
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: UnresolvedSaid
Observation
“The first example illustrates that ChatGPT with DALL-E 3 is unable to reliably associate labels correctly with the parts of an object.”
Our reading: ChatGPT with DALL-E 3 cannot reliably associate labels correctly with the parts of an object.
No horizon Evidence as of
Limitation: Unresolved does not mean the bicycle example was correct.
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: UnresolvedSaid
Observation
“In particular, DALL-E2 was very poor at following specifications about parts of objects.”
Our reading: As of October 2022, DALL-E2 was very poor at following prompt specifications about parts of objects.
No horizon Evidence as of
Limitation: This is an evidence gap, not evidence that DALL-E2 performed well.
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: UnresolvedSaid
Observation
“We did not see anything like the kind of improvement that ChatGPT has exhibited over math word problems, over the last two years.”
Our reading: In the authors' body-parts image experiments, they did not see anything like the kind of improvement that ChatGPT has shown on math word problems over the last two years.
No horizon Evidence as of
Limitation: Related benchmark gains cannot substitute for the complete comparison.
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: UnresolvedSaid
Observation
“The second example illustrates that it cannot reliably produce a non-standard number of body parts.”
Our reading: ChatGPT with DALL-E 3 cannot reliably produce images with a non-standard number of body parts.
No horizon Evidence as of
Limitation: No verified contemporary prompt-and-image evaluation directly measures requested nonstandard anatomical counts.
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: UnresolvedSaid
Observation
“This just in, right on the heels of Satya Nadella also publicly acknowledging diminishing returns:”
Our reading: Satya Nadella has publicly acknowledged diminishing returns.
No horizon Evidence as of
Limitation: Acknowledging possible limits could support a weaker interpretation of Marcus's wording.
Assessment: UnresolvedSaid
Observation
“Two years on, on the most important question of all – factuality and reliability — we are still pretty much where we were when ChatGPT first came out: wishing and hoping.”
Our reading: Two years after ChatGPT came out, on factuality and reliability, which he calls the most important question, we are still pretty much where we were at its release: wishing and hoping.
No horizon Evidence as of
Limitation: Neither study establishes aggregate progress from November 2022 to November 2024 across representative factuality and reliability tasks.
Source: ChatGPT, at Age Two
Assessment: UnresolvedSaid
Observation
“There has been a ton of experimentation, but relatively few well-documented success stories, and some fairly negative stories like a recent one at Business Insider on GPT-powered CoPilot,”
Our reading: There has been much experimentation with ChatGPT-style AI but relatively few well-documented success stories.
No horizon Evidence as of
Limitation: The survey measures deployment and measurement practices, not the number of public success stories.
Source: ChatGPT, at Age Two
Assessment: UnresolvedSaid
Observation
“But a lot of of theories about how ChatGPT would be used in practice have fizzled out.”
Our reading: Many theories about how ChatGPT would be used in practice have fizzled out.
No horizon Evidence as of
Limitation: The population of predictions is undefined.
Source: ChatGPT, at Age Two
Assessment: UnresolvedSaid
Observation
“Yet hundreds of billions more have been invested on further speculation that scaling would somehow magically cure problems that actually appear to be inherent with the technology.”
Our reading: Hundreds of billions more have been invested on further speculation that scaling would somehow magically cure problems that appear to be inherent to the technology.
No horizon Evidence as of
Limitation: The spending categories cannot be added without checking overlap and accounting treatment.
Source: ChatGPT, at Age Two
Assessment: UnresolvedSaid
Observation
“But things have been changing rapidly, and the great scaling retrenchment has already begun.”
Our reading: As of November 2024, a major retreat from scaling claims in the AI field has already begun.
No horizon Evidence as of
Limitation: Great retrenchment has no stated threshold.
Assessment: UnresolvedSaid
Observation
“The new version is that there is not one scaling law, but three: scaling with how long you train a model (which isn’t really holding anymore), scaling with how long you post-train a model, and scaling with how long you let a given model wrestle with a given problem (or what Satya Nadella called scaling with “inference time compute”).”
Our reading: Scaling performance by training models longer is no longer really holding.
No horizon Evidence as of
Limitation: The strongest experimental counterevidence concerns an earlier model and is context only.