Claims that can’t be checked by Gary Marcus, page 12 of 24
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Claims that can’t be checked by Gary Marcus (2353)
Public evidence cannot settle these claims, so they are recorded and attributed, not scored. Most were screened out: both AI providers read the claim on its own and found it a value judgment, rhetoric, hedged, too vague to test, or about something no public evidence could show. The rest were found to be recommendations or too open to test in a full review.
Page 12 of 24: claims 1101 to 1200 of 2353, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Building $500B worth of power and data centers in the service of enormous collections of those chips isn't looking so sensible, either.”
Our reading: Building $500B worth of power and data centers to serve enormous collections of top Nvidia chips isn't looking so sensible either.
No horizon
Source: The race for "AI Supremacy" is over — at least for now.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Furthermore, as I have argued for years, LLMs are too opaque, too unwieldy, and too difficult to debug and verify. The answer lies elsewhere; betting our future on that single idea is foolish.”
Our reading: LLMs are too opaque, unwieldy and hard to debug and verify, so betting the future on them is foolish.
No horizon
Source: The race for "AI Supremacy" is over — at least for now.
Assessment: Can’t be checkedSaid
Causal claim
“And maybe kneecapped our greatest silicon company, Nvidia.”
Our reading: US export controls may have kneecapped Nvidia.
No horizon
Source: The race for "AI Supremacy" is over — at least for now.
Assessment: Can’t be checkedSaid
Forecast
“Nvidia could soon take a serious hit, too, for two reasons. First, the DeepSeek results suggest that large language models might be trained far more efficiently going forward.”
Our reading: Nvidia could soon take a serious hit because LLMs may be trainable far more efficiently.
No horizon
Qualifier: LLMs can be trained far more efficiently
Source: The race for "AI Supremacy" is over — at least for now.
Assessment: Can’t be checkedSaid
Observation
“Like a lot of what Sam says, this is based on the conjecture, or in this case multiple conjectures:”
Our reading: Altman's claim that Stargate will be great for the country rests on several conjectures, as much of what he says does.
No horizon
Source: Hot take on an AI catfight
Assessment: Can’t be checkedSaid
Observation
“The equally speculative conjecture that the U.S. will long term have an advantage over China if the U.S. builds the hypothesized infrastructure, as if China isn’t trying to build the same.”
Our reading: The idea that building the infrastructure will give the U.S. a long-term AI advantage over China is speculative, since China is trying to build the same.
No horizon
Qualifier: if the U.S. builds the infrastructure
Source: Hot take on an AI catfight
Assessment: Can’t be checkedSaid
Observation
“The entirely speculative conjecture that LLMs or something else OpenAI figures out how to be build will be enormously profitable.”
Our reading: The idea that LLMs or other OpenAI technology will be enormously profitable is entirely speculative.
No horizon
Source: Hot take on an AI catfight
Assessment: Can’t be checkedSaid
Observation
“Elon thinks Sam is playing the same move that they once played together. He could be right.”
Our reading: Musk could be right that Altman is overstating the Stargate funding, the same way OpenAI's original $1B funding claim was overstated.
No horizon
Source: Hot take on an AI catfight
Assessment: Can’t be checkedSaid
Observation
“The entirely speculative conjecture that any profits will actually do much to help the American people, as opposed to just enriching those who own that infrastructure.”
Our reading: The idea that any AI infrastructure profits will much help the American people, rather than just enrich the owners, is entirely speculative.
No horizon
Source: Hot take on an AI catfight
Assessment: Can’t be checkedSaid
Rhetoric
“Sam Altman himself hinted we were close, just two weeks ago, writing “We are now confident we know how to build AGI as we have traditionally understood it”, which to be honest gave me Elizabeth Holmes vibes.”
Our reading: Marcus says Altman's statement that OpenAI is confident it knows how to build AGI gave him, to be honest, Elizabeth Holmes vibes.
No horizon
Qualifier: personal impression ('gave me ... vibes')
Assessment: Can’t be checkedSaid
Forecast
“But not this year (or next, or the one after that, and probably not this decade, except in narrow use cases).”
Our reading: AI agents probably will not do a huge fraction of human knowledge work this decade, except in narrow use cases.
No horizon
Qualifier: except in narrow use cases
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“A system answering a river-crossing problem should not answer by superficial analogy to the words in similar problems found in a giant scrape of the internet, but rather by reasoning through concepts about boats, rivers, people, and so on.”
Our reading: AI systems should solve problems by reasoning over concepts, not by surface analogy to text scraped from the internet.
No horizon
Assessment: Can’t be checkedSaid
Rhetoric
“Anyone who tries to tell you otherwise is either confused or trying to hype something.”
Our reading: Anyone claiming AGI has been achieved is either confused or hyping something.
No horizon
Assessment: Can’t be checkedSaid
Observation
“but that regurgitation is an illusion. Genuine depth is lacking.”
Our reading: The depth LLMs seem to have at times, from regurgitating or replicating deep analyses found on the internet, is an illusion; genuine depth is lacking.
No horizon
Qualifier: at times (per Mahabal's quoted observation)
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Moving past broad but shallow AI and into a new and hopefully safer and more reliable regime should be a major research priority for the human species.”
Our reading: Humanity should make it a major research priority to move beyond broad, shallow AI toward safer, more reliable AI.
No horizon
Assessment: Can’t be checkedSaid
Causal claim
“Counting on AI that is shallow and not reliable, it should also be said, could cause huge problems.”
Our reading: Relying on shallow, unreliable AI could cause huge problems.
No horizon
Assessment: Can’t be checkedSaid
Conditional forecast
“Then, and only then, will we have reliable solutions to the unreliability that has plagued recent AI.”
Our reading: AI's unreliability will be reliably solved only once systems reason over concepts with deep understanding.
No horizon
Only if: only if systems achieve deep conceptual understanding
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“For all these reasons, I propose that we call LLMs broad, shallow intelligence..”
Our reading: Marcus proposes that LLMs should be called 'broad, shallow intelligence'.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Instead, LLMs are still unpredictable enough that humans should still be in the loop in most case.”
Our reading: Because LLMs remain unpredictable, humans should stay in the loop in most cases.
No horizon
Qualifier: in most cases
Assessment: Can’t be checkedSaid
Observation
“We can never guarantee that broad but shallow AI will be safe, or that it will be human-compatible (in the words of Stuart Russell), or that it will do what we ask of it.”
Our reading: Broad but shallow AI can never be guaranteed to be safe, human-compatible, or to do what we ask.
No horizon
Assessment: Can’t be checkedSaid
Observation
“Every declaration of us already reaching AGI I have seen rests on redefining the term downwards, as I discussed last week,”
Our reading: Every claim he has seen that AGI has already been reached depends on lowering the definition of AGI.
No horizon Evidence as of
Limitation: The evidence does not show which declarations Marcus saw. It also does not show whether he saw these two.
Assessment: Can’t be checkedSaid
Observation
“Nobody serious thinks that we have reached AGI (artificial general intelligence) yet.”
Our reading: No serious person thinks that AGI (artificial general intelligence) has been reached yet.
No horizon
Scope: among people he counts as serious
Assessment: Can’t be checkedSaid
Observation
“The river crossing example and many others shows that LLMs often use the words without a deep understanding what those words mean.”
Our reading: LLMs often use words without deeply understanding what they mean.
No horizon
Assessment: Can’t be checkedSaid
Observation
“o3 is a real advance in some domains beyond GPT-4, but it’s likely not the quantum leap forward people would expect of the name GPT-5, and at a rumored $1000/querry certainly not at the price point anyone might have hoped for”
Our reading: o3 is a real advance beyond GPT-4 in some domains, but it is likely not the quantum leap people would expect of the name GPT-5, and at a rumored $1000 per query it is certainly not at the price point anyone might have hoped for.
No horizon
Scope: in some domains
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Gary Marcus wishes the media would hold those who make unrealistic promises to account.”
Our reading: The media should hold those who make unrealistic AI promises to account.
No horizon
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: Can’t be checkedSaid
Forecast
“We might see GPT-5 this year, but given bottlenecks on data and inherent limits in LLMs we may well not.”
Our reading: GPT-5 may well not be released in 2025, owing to data bottlenecks and inherent limits of LLMs.
No horizon
Qualifier: given bottlenecks on data and inherent limits in LLMs
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: Can’t be checkedSaid
Forecast
“He made some of his usual overpromises (500,000 humanoid robots in three years, which seems to very unlikely)”
Our reading: Musk's promise of 500,000 humanoid robots in three years seems very unlikely to be fulfilled.
No horizon
Qualifier: seems
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: Can’t be checkedSaid
Forecast
“I won’t be at all surprised if don’t even see anything worthy of the GPT-5 name”
Our reading: It is quite possible that no model worthy of the GPT-5 name will appear in 2025.
No horizon
Source: AGI isn’t coming in 2025, and GPT-5 probably isn’t either.
Assessment: Can’t be checkedSaid
Observation
“There are some applications where the current systems are reliable enough to be practically useful, though not nearly as many as is often claimed.”
Our reading: Current AI systems are reliable enough to be useful in some applications, but far fewer than often claimed.
No horizon Evidence as of
Limitation: The claim identifies no advertised comparison set, common unit for counting applications, or standard for practical reliability.
Source: The Five Stages of AGI Grief
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“If a lot of people’s goalposts have moved, mine have not. same as before, I will be satisfied if and when we reach AI systems are “flexible and general, with resourcefulness and reliability comparable to (or beyond) human intelligence” – and not until.”
Our reading: Marcus's goalposts for AGI have not moved: he will be satisfied if and when AI systems are flexible and general, with resourcefulness and reliability comparable to or beyond human intelligence, and not until.
No horizon
Qualifier: if and when AI systems are flexible and general, with resourcefulness and reliability comparable to (or beyond) human intelligence
Source: The Five Stages of AGI Grief
Assessment: Can’t be checkedSaid
Observation
“In reality, the definition of AGI, while not perfect, is reasonably clear — and has been reasonably clear — for a long time.”
Our reading: The definition of AGI, though imperfect, is reasonably clear and has been for a long time.
No horizon
Source: The Five Stages of AGI Grief
Assessment: Can’t be checkedSaid
Observation
“Current models trained on next-token prediction have remarkable abilities and remarkable weaknesses. The scope of those abilities and weaknesses of the current systems is not well understood.”
Our reading: Current next-token prediction models have remarkable abilities and weaknesses, whose scope is not well understood.
No horizon
Source: The Five Stages of AGI Grief
Assessment: Can’t be checkedSaid
Observation
“But so far I have not seen a principled solution to even one of these longstanding problems, and that tells me a lot.”
Our reading: So far Marcus has not seen a principled solution to even one of these longstanding problems, and he says that tells him a lot.
No horizon
Qualifier: so far
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: Can’t be checkedSaid
Forecast
“[Note also that the current idea, test-time “scaling”, is very expensive, and will likely prove unreliable out of domains where augmentation is adequate, in which case it will get comparatively little commercial traction.”
Our reading: Test-time scaling is very expensive and will likely prove unreliable outside domains where data augmentation is adequate.
No horizon
Qualifier: outside domains where augmentation is adequate
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: Can’t be checkedSaid
Observation
“In domains where that’s not possible, o1 may not even be that much better than GPT-4 (e.g on some language tasks), as you can see if you read OpenAI’s reports carefully.”
Our reading: Where data augmentation isn't possible, o1 may not be much better than GPT-4, for example on some language tasks.
No horizon
Qualifier: in domains where data augmentation isn't possible
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: Can’t be checkedSaid
Observation
“o1 probably isn’t as robust as people might have hoped. Even on certain benchmarks, results may be fragile or difficult to replicate:”
Our reading: o1 is probably less robust than hoped, and some benchmark results may be fragile or hard to replicate.
No horizon
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: Can’t be checkedSaid
Observation
“In o1’s best performances cases, there may have been heavy data augmentation that is not possible in more open-ended domains.”
Our reading: In o1's best-performance cases there may have been heavy data augmentation that is not possible in more open-ended domains.
No horizon
Scope: in o1's best-performance cases
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: Can’t be checkedSaid
Observation
“Here is a thread of links to a several recent observations that I think show how far we are from robust intelligence.”
Our reading: Marcus thinks several recent observations show how far AI is from robust intelligence.
No horizon
Qualifier: I think
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: Can’t be checkedSaid
Observation
“I do not share his confidence.”
Our reading: Marcus does not share Sam Altman's confidence that OpenAI knows how to build AGI.
No horizon
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: Can’t be checkedSaid
Forecast
“The presentation of o3 was problematic, and no clean test of out-of-domain generalization was provided, suggesting that we may see the kinds of problems with distribution shift as ever, outside of semi-closed domains where synthetic data can readily be generated.”
Our reading: Marcus says the presentation of o3 was problematic and gave no clean test of out-of-domain generalization, which suggests o3 may show the same distribution-shift problems as ever outside semi-closed domains where synthetic data can readily be generated.
No horizon
Scope: outside semi-closed domains where synthetic data can readily be generated
Source: Why I don’t share Sam Altman’s confidence that AGI is basically a solved problem
Assessment: Can’t be checkedSaid
Causal claim
“But fundamentally its videos are more like visual collages than the renderings of game engines, which is why they sometimes have a surreal or magic trick quality.”
Our reading: Sora's videos are fundamentally more like visual collages than game-engine renderings, which is why they sometimes have a surreal or magic-trick quality.
No horizon
Qualifier: more like (comparative, not categorical)
Assessment: Can’t be checkedSaid
Conditional forecast
“In our view, it is only once AI researchers grapple directly with the challenging problems inherent in commonsense reasoning about entities that interact in space and persist in time that the field of general-purpose AI will finally begin to mature.”
Our reading: General-purpose AI will begin to mature only once researchers directly tackle commonsense reasoning about entities interacting in space and persisting in time.
No horizon
Only if: only once researchers grapple directly with commonsense reasoning about entities in space and time
Assessment: Can’t be checkedSaid
Forecast
“My essay was about the small but very real possibility that 2025 could see the worst cyberattack in history, perhaps catalyzed in some fashion by generative AI (which was #24 in my list of 25 AI predictions for 2025).”
Our reading: There is a small but real possibility that 2025 sees the worst cyberattack in history, possibly catalyzed in some way by generative AI.
No horizon
Qualifier: small but very real possibility
Assessment: Can’t be checkedSaid
Forecast
“Technical moat will continue to be elusive. Instead, there will be more convergence on broadly similar models, across both US and China; some systems in Europe will catch up to roughly the same place.”
Our reading: Filed under 'Medium confidence': through the end of 2025, a technical moat will continue to be elusive; instead there will be more convergence on broadly similar models across both the US and China, and some systems in Europe will catch up to roughly the same place.
Deadline passed Evidence as of
Limitation: This record does not show what happened in 2025. No evidence about the gap between leading models is part of this record.
Assessment: Can’t be checkedSaid
Forecast
“Few companies (and even fewer consumers) will adopt o3 at wide scale because of concerns about price and robustness relative to that price.”
Our reading: Filed under 'Medium confidence': by the end of 2025, few companies (and even fewer consumers) will adopt OpenAI's o3 at wide scale, because of concerns about price and robustness relative to that price.
Deadline passed Evidence as of
Limitation: This record does not show how widely o3 was adopted in 2025. No evidence on its adoption is part of this record.
Assessment: Can’t be checkedSaid
Forecast
“Sora will continue to have trouble with physics.”
Our reading: Filed under 'Medium confidence': at the end of 2025, OpenAI's Sora will still have trouble with physics.
Deadline passed Evidence as of
Limitation: The only evidence shown is OpenAI's own description, not an independent test.
Assessment: Can’t be checkedSaid
Forecast
“2025 could well be the year in which valuations for major AI companies start to fall.”
Our reading: Valuations of major AI companies could start to fall in 2025.
No horizon
Qualifier: filed under medium confidence
Assessment: Can’t be checkedSaid
Forecast
“Companies will continue to experiment with AI, but adoption to production-grade systems scaled out in the real-world will continue to be tentative.”
Our reading: With medium confidence, by the end of 2025 companies will continue to experiment with AI, but adoption of production-grade systems scaled out in the real world will continue to be tentative.
No horizon
Qualifier: filed under medium confidence
Assessment: Can’t be checkedSaid
Forecast
“Instead we may see models like o1 that are quite good at many tasks for which high-quality synthetic data can be created, but in other domains only incrementally better than GPT-4.”
Our reading: In 2025 we may see o1-like models good at synthetic-data-friendly tasks but only incrementally better than GPT-4 elsewhere.
No horizon
Qualifier: tasks where high-quality synthetic data can be created
Assessment: Can’t be checkedSaid
Observation
“My 2023 prediction that OpenAI might someday be seen as the WeWork of AI has not been confirmed, but they are far from out of the woods.”
Our reading: His 2023 prediction that OpenAI might someday be seen as the WeWork of AI has not been confirmed, but OpenAI is far from out of the woods.
No horizon Evidence as of
Limitation: This finding does not establish that OpenAI was financially secure or that the concern was unreasonable.
Assessment: Can’t be checkedSaid
Forecast
“Sora will continue to have trouble with physics.”
Our reading: With medium confidence, by the end of 2025 Sora will continue to have trouble with physics.
No horizon
Qualifier: filed under medium confidence
Assessment: Can’t be checkedSaid
Forecast
“Few companies (and even fewer consumers) will adopt o3 at wide scale because of concerns about price and robustness relative to that price.”
Our reading: With medium confidence, by the end of 2025 few companies, and even fewer consumers, will adopt o3 at wide scale because of concerns about price and robustness relative to that price.
No horizon
Qualifier: because of concerns about price and robustness relative to that price
Assessment: Can’t be checkedSaid
Observation
“Long term security, especially at high valuations, is this far from assured.”
Our reading: OpenAI's long-term security, especially at high valuations, is far from assured.
No horizon
Qualifier: especially at high valuations
Assessment: Can’t be checkedSaid
Forecast
“Technical moat will continue to be elusive. Instead, there will be more convergence on broadly similar models, across both US and China; some systems in Europe will catch up to roughly the same place.”
Our reading: With medium confidence, by the end of 2025 a technical moat will continue to be elusive, with more convergence on broadly similar models across the US and China, and some European systems catching up to roughly the same place.
No horizon
Qualifier: filed under medium confidence
Assessment: Can’t be checkedSaid
Forecast
“Neurosymbolic AI will become much more prominent.”
Our reading: With medium confidence, by the end of 2025 neurosymbolic AI will become much more prominent.
No horizon
Qualifier: filed under medium confidence
Assessment: Can’t be checkedSaid
Forecast
“o3 has been previewed but not released likely will likely be quite expensive.)”
Our reading: OpenAI's o3 will likely be quite expensive when released.
No horizon
Assessment: Can’t be checkedSaid
Forecast
“We may well see a large-scale cyberattack in which Generative AI plays an important causal role, perhaps in one of the four ways discussed in a short essay of mine that will appear shortly in Politico.”
Our reading: In 2025 there may be a large-scale cyberattack in which generative AI plays an important causal role.
No horizon
Qualifier: filed under low confidence
Assessment: Can’t be checkedSaid
Observation
“Commercial artists and voiceover actors have perhaps been the hardest hit so far.”
Our reading: Commercial artists and voiceover actors have perhaps been hardest hit by AI so far.
No horizon
Assessment: Can’t be checkedSaid
Forecast
“I wouldn’t be shocked if none were reliably solved by the end of the year.”
Our reading: Possibly none of the Marcus-Brundage tasks will be reliably solved by end of 2025.
No horizon
Assessment: Can’t be checkedSaid
Forecast
“(Google’s Veo 2 seems to be better but I have not been able to experiment with it, and suspect that changes of state and the persistence of objects will still cause problems;”
Our reading: Marcus suspects, without having tested it, that changes of state and the persistence of objects will still cause problems for Google's Veo 2, though it seems better than Sora.
No horizon
Qualifier: has not been able to experiment with it
Assessment: Can’t be checkedSaid
Forecast
“There could continue to be no “GPT-5 level” model (meaning a huge, across the board quantum leap forward as judged by community consensus) throughout 2025.”
Our reading: There could be no GPT-5 level model, meaning a huge across-the-board leap by community consensus, throughout 2025.
No horizon
Qualifier: filed under low confidence
Assessment: Can’t be checkedSaid
Forecast
“AI will presumably solve all of these someday, of course, but current techniques have thus far not been adequate.”
Our reading: AI will presumably eventually solve compositionality, factuality and reasoning, but current techniques have not been adequate.
No horizon
Qualifier: someday
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Both of us agree that the achievement of 8 out of 10 of these tasks would be strong evidence for the G (General) in Artificial General Intelligence having been achieved.”
Our reading: Marcus and Brundage agree that achieving 8 of the 10 listed tasks would be strong evidence that the 'General' in Artificial General Intelligence had been achieved.
No horizon
Qualifier: jointly agreed with Miles Brundage
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Because of the pretraining, and lack of comparability, what was actually shown was disappointing.”
Our reading: Because of the pretraining and the lack of comparability with humans, what o3 actually showed on ARC was disappointing.
No horizon
Qualifier: because of the pretraining and lack of comparability
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“The media should be asking hard questions, not fanning hype.”
Our reading: The media should ask hard questions about AI instead of fanning hype.
No horizon
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: Can’t be checkedSaid
Conditional forecast
“Until there is considerable external scientific scrutiny (so far there has been none), we won’t really know exactly what the o3 advance is or how important it is.”
Our reading: Until o3 receives considerable external scientific scrutiny (so far it has had none), we won't really know exactly what the o3 advance is or how important it is.
No horizon
Only if: until there is considerable external scientific scrutiny
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“As the scientist Adan Becerra, PhD put it (and Chollet publicly agreed) the best thing would have been to present data for the “base model” without the pretraining.”
Our reading: OpenAI should have presented ARC results for the o3 base model without the pretraining.
No horizon
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“The video should have been much clearer about what was actually tested and what was actually trained.”
Our reading: OpenAI's o3 announcement video should have been much clearer about what was tested and what was trained.
No horizon
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“As NYU prof Brenden Lake pointed out, the test should never have been called ARC-AGI.”
Our reading: The ARC test should never have been named ARC-AGI.
No horizon
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“People in the media probably shouldn’t even joke about o3 being AGI.”
Our reading: People in the media probably shouldn't even joke about o3 being AGI.
No horizon Evidence as of
Limitation: Only this normative sentence is assessed. Nearby factual claims in the post are not covered.
Source: 𝗼𝟯 “𝗔𝗥𝗖 𝗔𝗚𝗜” 𝗽𝗼𝘀𝘁𝗺𝗼𝗿𝘁𝗲𝗺 𝗺𝗲𝗴𝗮𝘁𝗵𝗿𝗲𝗮𝗱: 𝘄𝗵𝘆 𝘁𝗵𝗶𝗻𝗴𝘀 𝗴𝗼𝘁 𝗵𝗲𝗮𝘁𝗲𝗱, 𝘄𝗵𝗮𝘁 𝘄𝗲𝗻𝘁 𝘄𝗿𝗼𝗻𝗴, 𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗮𝗹𝗹 𝗺𝗲𝗮𝗻𝘀
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Spending a quarter trillion dollars in order to have something 1000 times more expensive than average humans is maybe not the way to go.”
Our reading: Spending a quarter trillion dollars to get systems 1000x more expensive than humans is maybe not the right approach.
No horizon
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Until lots of people get to try o3 on many different kinds of tasks, we should not assume that it is reliable.”
Our reading: People should not assume o3 is reliable until many people have tested it on many kinds of tasks.
No horizon
Qualifier: until many people test it on many kinds of tasks
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: Can’t be checkedSaid
Forecast
“I wish that others would give lines in the sand in advance; instead, they are likely to craft definitions post hoc, or not give them at all.”
Our reading: Those claiming AGI are likely to craft definitions post hoc or give none at all.
No horizon
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“And I am saying, as every roboticist on the planet knows, that we shouldn’t take demos seriously, until purported products are actually released and subjected to outside scrutiny.”
Our reading: AI demos should not be taken seriously until the products are released and subjected to outside scrutiny.
No horizon
Qualifier: until products are released and externally scrutinized
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: Can’t be checkedSaid
Forecast
“Furthermore, the impact of all this on the environment could be immense.”
Our reading: The environmental impact of such high-compute AI systems could be immense.
No horizon
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: Can’t be checkedSaid
Observation
“As in O3 may work for problems where massive data augmentation is possible, but not in others where it’s not.”
Our reading: o3 may work on problems allowing massive data augmentation but not on problems that do not.
No horizon
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: Can’t be checkedSaid
Observation
“I couldn’t agree more. Yesterday’s demo, impressive as it was, doesn’t change that.”
Our reading: The o3 demo does not change the view that AGI requires true logical reasoning rather than statistical tricks.
No horizon
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: Can’t be checkedSaid
Observation
“Nothing in the o3 presentation yesterday suggested to me that this decades-old core problem has been remotely solved.”
Our reading: Nothing in the o3 presentation suggested to Marcus that the decades-old core problem of neural networks generalizing better within distribution than outside it has been remotely solved.
No horizon Evidence as of
Limitation: This outcome makes no judgment about whether the presentation established a general solution to distribution shift.
Source: o3, AGI, the art of the demo, and what you can expect in 2025
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“The problems by now seem clear. Suchir was right to consider them.”
Our reading: The copyright problems by now seem clear, and Suchir Balaji was right to consider them.
No horizon
Qualifier: seem
Source: Generative AI’s Continuing Copyright Problems, an Essay in Memory of Suchir Balaji, 1998 - 2024
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Society needs to do more to protect people like Suchir Balaji.”
Our reading: Society should do more to protect AI whistleblowers like Suchir Balaji.
No horizon
Source: Generative AI’s Continuing Copyright Problems, an Essay in Memory of Suchir Balaji, 1998 - 2024
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“It is unfortunate that CA SB-1047’s whistleblower protections were vetoed.”
Our reading: It is unfortunate that the whistleblower protections in California SB-1047 were vetoed.
No horizon
Source: Generative AI’s Continuing Copyright Problems, an Essay in Memory of Suchir Balaji, 1998 - 2024
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“I wouldn’t take seriously any of its anthropomorphic language about LLMs “knowing” or having scheming or having intentions to deceive, etc.”
Our reading: Anthropomorphic language about LLMs knowing, scheming, or intending to deceive should not be taken seriously.
No horizon
Qualifier: in the Apollo Research scheming paper
Source: Humanity’s “Oh shit!” AI moment?
Assessment: Can’t be checkedSaid
Causal claim
“They more power we give them (but hooking up them to agents, people, etc), the more risk we are incurring.”
Our reading: Giving LLMs more power, such as hooking them up to agents, increases risk.
No horizon Evidence as of
Limitation: The only evidence item is one company's report on misuse of its own models. It records incidents and measures nothing about how harm scales with the power granted.
Source: Humanity’s “Oh shit!” AI moment?
Assessment: Can’t be checkedSaid
Forecast
“Even if machines don’t ever rise up a la the Terminator, a bad actor could still leverage LLMs to cause massive mayhem. Possibly soon.”
Our reading: A bad actor could leverage LLMs to cause massive mayhem, possibly soon.
No horizon
Qualifier: by a bad actor
Source: Humanity’s “Oh shit!” AI moment?
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“The important thing is that paper makes it very clear that nobody should ever take LLMs at their word.”
Our reading: Nobody should ever take LLMs at their word.
No horizon
Source: Humanity’s “Oh shit!” AI moment?
Assessment: Can’t be checkedSaid
Causal claim
“The longer we delay in regulation, the more risk we are incurring.”
Our reading: Delaying AI regulation increases the risk society incurs.
No horizon Evidence as of
Limitation: The record has no outcome evidence at all, either for or against the claim.
Source: Humanity’s “Oh shit!” AI moment?
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“There’s a lot of anthropomorphism here in this paper that I find hard to take, but it is in other ways a smart and carefully considered paper — and ought to be a wakeup call.”
Our reading: Apollo Research's paper on in-context scheming contains a lot of anthropomorphism he finds hard to take, but is in other ways a smart and carefully considered paper and ought to be a wakeup call.
No horizon
Qualifier: in other ways
Source: Humanity’s “Oh shit!” AI moment?
Assessment: Can’t be checkedSaid
Causal claim
“That may leave all of us extremely vulnerable to this mess.”
Our reading: That Congress achieved little and that anti-regulation AI people like Andreessen are having a good run may leave all of us extremely vulnerable to this mess.
No horizon
Qualifier: may
Source: Humanity’s “Oh shit!” AI moment?
Assessment: Can’t be checkedSaid
Causal claim
“I don’t doubt this, when stated so vaguely, but you can’t improve the engineering if you can’t identify the problems and weak points, such that you can address them.”
Our reading: You can't improve the engineering if you can't identify its problems and weak points so that you can address them.
No horizon Evidence as of
Limitation: This does not establish that the maxim is true or false.
Source: Casey (Newton) Strikes Out
Assessment: Can’t be checkedSaid
Forecast
“But ignoring all the evidence that suggests that we won’t be there in the very near term does a disservice to his pretense towards objectivity.”
Our reading: Evidence suggests superintelligence will not be reached in the very near term.
No horizon
Qualifier: per available evidence
Source: Casey (Newton) Strikes Out
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Our best defense against the possible dangers of superintelligence must begin with a realistic appraisal of where we are and how much time we have.”
Our reading: Defending against superintelligence dangers must start with a realistic appraisal of current progress and timelines.
No horizon
Source: Casey (Newton) Strikes Out
Assessment: Can’t be checkedSaid
Forecast
“A massive investment of hundreds of billions is unlikely to pay off.”
Our reading: The massive investment of hundreds of billions of dollars in AI scaling is unlikely to pay off.
No horizon
Source: Casey (Newton) Strikes Out
Assessment: Can’t be checkedSaid
Observation
“The John Locke hypothesis that you can learn physics purely from sense data is failing, over, and over, and over again; this time perhaps (if I had to guess) to the tune of a billion dollars.”
Our reading: The author guesses that Sora's failed attempt to learn physics from data may have cost around a billion dollars.
No horizon
Qualifier: author's guess
Assessment: Can’t be checkedSaid
Causal claim
“One plausible explanation is that OpenAI has put a lot more work, with data augmentation and other such techniques, into math word problems than into image generation.”
Our reading: One plausible explanation for the difference in improvement is that OpenAI has put a lot more work, using data augmentation and other such techniques, into math word problems than into image generation.
No horizon
Qualifier: offered as one plausible explanation
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: Can’t be checkedSaid
Causal claim
“Relating language and the visual world robustly may require altogether new approaches to learning language.”
Our reading: Robustly relating language and the visual world may require altogether new approaches to learning language.
No horizon
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: Can’t be checkedSaid
Rhetoric
“Until then the foundations of AI remain unsteady.”
Our reading: Until new approaches robustly relate language and vision, the foundations of AI remain unsteady.
No horizon
Qualifier: until new approaches to learning language exist
Source: Don’t Ride This Bike! Generative AI’s persistent trouble with compositionality and parts
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“AI needs substantive regulation.”
Our reading: AI needs substantive regulation.
No horizon
Source: Hard-forked! Casey Newton’s distorted portrait of Gary Marcus and AI skepticism
Assessment: Can’t be checkedSaid
Forecast
“Many downsides are already heere, and potential misuse in the military seems imminent.”
Our reading: Military misuse of LLMs seems imminent, adding to downsides such as covert racism, deepfakes and discrimination that already exist.
No horizon
Source: Hard-forked! Casey Newton’s distorted portrait of Gary Marcus and AI skepticism
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“As AI comes to have more power in society, AI skepticism becomes an increasingly important counterweight, and we should all want AI skepticism to be done right.”
Our reading: As AI gains power, AI skepticism becomes an increasingly important counterweight and should be done well.
No horizon
Qualifier: as AI gains more power in society
Source: Hard-forked! Casey Newton’s distorted portrait of Gary Marcus and AI skepticism
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“We ought to prepare – now – for AI to get a lot more dangerous than it already is.”
Our reading: Society should prepare now for AI becoming much more dangerous than it already is.
No horizon
Source: Hard-forked! Casey Newton’s distorted portrait of Gary Marcus and AI skepticism