Claims that can’t be checked by Gary Marcus, page 15 of 24
Deadlines are judged as of 2026-10-29. About Gary Marcus and the sources covered.
Claims that can’t be checked by Gary Marcus (2353)
Public evidence cannot settle these claims, so they are recorded and attributed, not scored. Most were screened out: both AI providers read the claim on its own and found it a value judgment, rhetoric, hedged, too vague to test, or about something no public evidence could show. The rest were found to be recommendations or too open to test in a full review.
Page 15 of 24: claims 1401 to 1500 of 2353, ordered by the date first said, newest first. To find a claim by its words, use search.
Assessment: Can’t be checkedSaid
Rhetoric
“Without independent scientists in the loop, with a real voice, we are lost.”
Our reading: Without independent scientists in the loop with a real voice, we are lost.
No horizon
Qualifier: without independent scientists in the loop, with a real voice
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: Can’t be checkedSaid
Forecast
“But OpenAI still have massive power, and with it a chance to screw up, and possibly put humanity at risk.”
Our reading: OpenAI has massive power and could possibly put humanity at risk.
No horizon
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: Can’t be checkedSaid
Observation
“Millions, perhaps billions, of documents likely have been taken without permission.”
Our reading: OpenAI has likely taken millions, perhaps billions, of documents without permission.
No horizon
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“If the first lesson is specific - that OpenAI should not be fully trusted – the second lesson is general.”
Our reading: OpenAI should not be fully trusted.
No horizon
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: Can’t be checkedSaid
Observation
“The list above is not complete, but it shows a significant gap between the rosy rhetoric and actual reality, and in my view that gap has only grown over time, particularly over the last year or so..”
Our reading: The author's non-exhaustive list shows a significant gap between OpenAI's rosy rhetoric and reality, and in his view that gap has only grown over time, particularly over the last year or so.
No horizon
Qualifier: in my view
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“As I argue, in my upcoming book Taming Silicon Valley, our only real hope is if citizens speak up.”
Our reading: The only real hope for governing AI is citizens speaking up.
No horizon
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: Can’t be checkedSaid
Rhetoric
“Our government has been repeatedly complicit, treating Altman repeatedly as royalty.”
Our reading: The US government has repeatedly been complicit, repeatedly treating Altman as royalty.
No horizon
Source: What should we learn from OpenAI’s mistakes and broken promises?
Assessment: Can’t be checkedSaid
Forecast
“Partial regurgitation, no matter how fluent, does not, and will not ever, constitute genuine comprehension.”
Our reading: Partial regurgitation, no matter how fluent, does not and will not ever constitute genuine comprehension.
No horizon
Qualifier: no matter how fluent
Assessment: Can’t be checkedSaid
Observation
“Neural nets do store weights, but that doesn’t mean that they know what they are talking about”
Our reading: Storing weights does not mean neural nets know what they are talking about.
No horizon
Assessment: Can’t be checkedSaid
Forecast
“Getting to real AI will require a different approach.”
Our reading: Getting to real AI will require a different approach from the partial-regurgitation approach of current LLMs.
No horizon
Assessment: Can’t be checkedSaid
Observation
“Instead, they appear to be focused precisely on financial return, and appear almost indifferent to some the ways in which their product has already hurt large numbers of people (artists, writers, voiceover actors, etc).”
Our reading: OpenAI appears to be focused precisely on financial return and appears almost indifferent to some of the ways its product has already hurt large numbers of people, such as artists, writers and voiceover actors.
No horizon
Qualifier: appear(s)
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“The real issue isn’t whether OpenAI would win in court, it’s what happens to all of us, if a company with a track record for cutting ethical corners winds up first to AGI.”
Our reading: The real issue is not whether OpenAI would win in court but what happens to all of us if a company with a track record for cutting ethical corners ends up first to AGI.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“And that we don’t leave our collective future entirely to self-regulation.”
Our reading: Society should not leave AI's future entirely to industry self-regulation.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“I pray that we can return to AI that is genuinely in the public interest.”
Our reading: AI development should go back to being genuinely in the public interest.
No horizon
Source: AI has become Sauron’s Ring
Assessment: Can’t be checkedSaid
Causal claim
“But the science has been corrupted by money.”
Our reading: The science of AI has been corrupted by money.
No horizon
Source: AI has become Sauron’s Ring
Assessment: Can’t be checkedSaid
Observation
“AI didn’t *have* to go down such a shitty path of enabling and accelerating surveillance capitalism, misinformation, nonconsensual deepfake porn, IP theft, etc.”
Our reading: AI has gone down a path of enabling and accelerating surveillance capitalism, misinformation, nonconsensual deepfake porn and IP theft, which was not inevitable.
No horizon
Source: AI has become Sauron’s Ring
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Artists and writer and actors don’t want their work to be used without their permission; if you want to use their stuff, you should compensate them, and get their permission.”
Our reading: Those who want to use artists', writers' and actors' work should compensate them and get their permission.
No horizon
Qualifier: if using creators' work
Source: The OpenAI board was right
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Gary Marcus has seen enough, and hopes that the new OpenAI board recognizes that Sam’s behavior is not consistent with what one would expect of a nonprofit that has pledged to help humanity.”
Our reading: The new OpenAI board should recognize that Sam Altman's behavior is inconsistent with a nonprofit pledged to help humanity.
No horizon
Source: The OpenAI board was right
Assessment: Can’t be checkedSaid
Observation
“Sam got away with a lot for a long time, but people are starting to see through the ruse.”
Our reading: People are starting to see through Sam Altman's ruse after he got away with a lot for a long time.
No horizon
Source: The OpenAI board was right
Assessment: Can’t be checkedSaid
Rhetoric
“Spin is a way of life at OpenAI; telling the truth is not.”
Our reading: OpenAI habitually spins rather than tells the truth.
No horizon
Source: The OpenAI board was right
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“People don’t leave companies with $86 billion valuations for nothing. Four key resignations thus far in 2024 is not a good sign.”
Our reading: The four key resignations from OpenAI so far in 2024 are not a good sign.
No horizon
Scope: thus far in 2024
Source: Chaos and tension at OpenAI
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“My friends in Washington should look into this.”
Our reading: Marcus's friends in Washington should look into the situation at OpenAI described in the post, including the resignations and the questions he raises about it.
No horizon
Scope: my friends in Washington
Source: Chaos and tension at OpenAI
Assessment: Can’t be checkedSaid
Observation
“As Elon Musk noted in his lawsuit against OpenAI, that mission seems to be in the rear view mirrors.”
Our reading: OpenAI's founding mission of benefiting humanity, with AI safety at its center, appears to have been left behind.
No horizon
Source: Chaos and tension at OpenAI
Assessment: Can’t be checkedSaid
Causal claim
“I don’t think it’s about a scary new algorithm per se (as many people have speculated) – I suspects it is about an attitude: the company’s attitude towards AI safety.”
Our reading: Marcus suspects Ilya Sutskever left OpenAI because of the company's attitude toward AI safety, not because of a scary new algorithm.
No horizon
Qualifier: regarding Ilya Sutskever's departure
Source: Chaos and tension at OpenAI
Assessment: Can’t be checkedSaid
Causal claim
“If OpenAI had GPT-5, they have would shown it.”
Our reading: If OpenAI had GPT-5, it would have shown it.
No horizon
Qualifier: if OpenAI had GPT-5
Source: Hot take on OpenAI’s new GPT-4o
Assessment: Can’t be checkedSaid
Causal claim
“OpenAI has presumably pivoted to new features precisely because they don’t know how produce the kind of capability advance that the “exponential improvement” would have predicted.”
Our reading: OpenAI has presumably pivoted to new features precisely because it does not know how to produce the kind of capability advance that 'exponential improvement' would have predicted.
No horizon
Qualifier: presumably
Source: Hot take on OpenAI’s new GPT-4o
Assessment: Can’t be checkedSaid
Observation
“Most importantly, each day in which there is no GPT-5 level model–from OpenAI or any of their well-financed, well-motivated competitors—is evidence that we may have reached a phase of diminishing returns.”
Our reading: The continued absence of a GPT-5 level model from OpenAI or competitors is evidence that LLM development may have reached a phase of diminishing returns.
No horizon
Qualifier: absence of any GPT-5 level model from OpenAI or competitors
Source: Hot take on OpenAI’s new GPT-4o
Assessment: Can’t be checkedSaid
Forecast
“Gary Marcus doesn’t know exactly what to make of all this, but feels like it could matter eventually. A lot.”
Our reading: The increasingly complicated Microsoft–OpenAI relationship could eventually matter a great deal.
No horizon
Source: Microsoft and OpenAI’s increasingly complicated relationship
Assessment: Can’t be checkedSaid
Causal claim
“In seeking to grab vast resources for an approach that is deeply flawed, Altman is diverting resources from alternative approaches from AI that might be more trustworthy, reliable, and interpretable, potentially incurring considerable environment costs, and distracting from better ways of using capital to help humanity more generally.”
Our reading: In seeking vast resources for a deeply flawed approach, Altman is diverting resources from alternative AI approaches that might be more trustworthy, reliable and interpretable, is potentially incurring considerable environmental costs, and is distracting from better ways of using capital to help humanity more generally.
No horizon
Qualifier: alternatives that might be more trustworthy, reliable, and interpretable
Source: The Sam Altman Playbook
Assessment: Can’t be checkedSaid
Conditional forecast
“If they lost one of those in court, and precedent against training on copyright materials was set, the entire enterprise could be in jeopardy.”
Our reading: If OpenAI lost a copyright lawsuit setting precedent against training on copyrighted materials, its entire enterprise could be in jeopardy.
No horizon
Only if: OpenAI loses a copyright lawsuit
Source: The Sam Altman Playbook
Assessment: Can’t be checkedSaid
Observation
“Many (except perhaps the humility and soft-spoken tone) are familiar from Elon Musk’s regular overpromising; almost all the AI companies are basically following the same playbook, albeit with less panache.”
Our reading: Almost all AI companies follow the same overpromising, speculative playbook as Altman.
No horizon
Source: The Sam Altman Playbook
Assessment: Can’t be checkedSaid
Rhetoric
“Altman isn’t out there selling a new product; he is selling hope.”
Our reading: Altman is selling hope rather than a new product.
No horizon
Source: The Sam Altman Playbook
Assessment: Can’t be checkedSaid
Forecast
“GoogleDeepMind, Anthropic, Meta or someone else could conceivably beat them there.”
Our reading: Google DeepMind, Anthropic, Meta or someone else could conceivably beat OpenAI to a GPT-5-level model.
No horizon
Qualifier: conceivably
Source: The Sam Altman Playbook
Assessment: Can’t be checkedSaid
Observation
“At Stanford he said, “I think every year for the next many we have dramatically more capable systems every year”, as if that were a sure thing, when of course it isn’t.”
Our reading: It is not a sure thing that AI systems will become dramatically more capable every year for many years.
No horizon
Source: The Sam Altman Playbook
Assessment: Can’t be checkedSaid
Forecast
“Meta will likely release a GPT-4 level model within months – for free.”
Our reading: Meta will likely release a free GPT-4-level model within months of May 2024.
No horizon
Qualifier: for free
Source: The Sam Altman Playbook
Assessment: Can’t be checkedSaid
Forecast
“Training GPT-4 was expensive, training GPT-5 maybe an order of magnitude more expensive. GPT-6 yet another order more expensive.”
Our reading: Training GPT-5 may cost an order of magnitude more than GPT-4, and GPT-6 another order of magnitude more.
No horizon
Source: The Sam Altman Playbook
Assessment: Can’t be checkedSaid
Observation
“Whether GenAI proves to be AGI is very much still up for grabs, and on any reasonable account we have a long way to go.”
Our reading: Whether generative AI proves to be AGI is very much still up for grabs, and on any reasonable account there is a long way to go.
No horizon
Qualifier: on any reasonable account
Source: The Sam Altman Playbook
Assessment: Can’t be checkedSaid
Forecast
“Social media might be an example. But things may actually get worse.”
Our reading: The epistemic situation may actually get worse than social media, which Marcus says might itself be an example of an epistemic clusterfuck.
No horizon
Qualifier: social media might be an example of an epistemic clusterfuck
Assessment: Can’t be checkedSaid
Causal claim
“Automatically amplifying the already inherent tendency towards extremism without fact-checking or reasoned analysis is a recipe for chaos.”
Our reading: Automatically amplifying the already inherent tendency towards extremism without fact-checking or reasoned analysis is a recipe for chaos.
No horizon
Qualifier: without fact-checking or reasoned analysis
Assessment: Can’t be checkedSaid
Conditional forecast
“But if the White House treats tech CEOs like rock stars, those rock stars are going to call the shots.”
Our reading: If the White House treats tech CEOs like rock stars, those CEOs will determine AI policy.
No horizon
Only if: if the White House treats tech CEOs like rock stars
Source: We can’t trust the fox to guard the henhouse, especially when it comes to AI
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Every one of us should be worried.”
Our reading: Every one of us should be worried about Washington remaining in Silicon Valley's thrall, elections under siege, widespread deepfake scams, and no AI law having passed.
No horizon
Scope: every one of us
Source: We can’t trust the fox to guard the henhouse, especially when it comes to AI
Assessment: Can’t be checkedSaid
Conditional forecast
“Schumer has had something like nine closed door meetings on AI, but (to my knowledge) yet to bring a single AI-related piece of legislation to the Senate floor. If he doesn’t, we are sitting ducks.”
Our reading: If Schumer does not bring AI-related legislation to the Senate floor, we will be sitting ducks, meaning highly vulnerable.
No horizon
Only if: if Schumer doesn't bring AI-related legislation to the Senate floor
Source: We can’t trust the fox to guard the henhouse, especially when it comes to AI
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Washington is supposed to be a counterweight, to keep the captain of industries honest. So far, they haven’t done enough.”
Our reading: So far, Washington has not done enough as a counterweight to keep the captains of industry honest.
No horizon
Qualifier: so far
Source: We can’t trust the fox to guard the henhouse, especially when it comes to AI
Assessment: Can’t be checkedSaid
Observation
“The premise of the book is that Silicon Valley is increasingly controlling our lives, and indeed the entire world, and that Washington (seat of the US government) is too much in their thrall to stand up tall enough against it.”
Our reading: Silicon Valley is increasingly controlling our lives and the entire world, and Washington is too much in its thrall to stand up tall enough against it.
No horizon
Scope: our lives, and indeed the entire world
Source: We can’t trust the fox to guard the henhouse, especially when it comes to AI
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“And that we need to act, now.”
Our reading: We need to act now to counter Silicon Valley's control.
No horizon
Source: We can’t trust the fox to guard the henhouse, especially when it comes to AI
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Particularly since Silicon Valley is prone to hype, Washington needs independent scientists in the room.”
Our reading: Because Silicon Valley is prone to hype, Washington needs independent scientists involved in AI policy discussions.
No horizon
Qualifier: since Silicon Valley is prone to hype
Source: We can’t trust the fox to guard the henhouse, especially when it comes to AI
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“One of the major themes of Taming Silicon Valley is the need for independent oversight; as the old proverb so rightly puts it, the fox can’t guard the henhouse.”
Our reading: AI needs independent oversight rather than self-regulation by the tech industry.
No horizon
Source: We can’t trust the fox to guard the henhouse, especially when it comes to AI
Assessment: Can’t be checkedSaid
Causal claim
“It’s almost impossible to envision the independence we might need when nobody in Washington knows where they will be come January, and pots of gold await in the Bay Area.”
Our reading: It is almost impossible to envision the independence we might need in Washington when nobody there knows where they will be come January and lucrative jobs await in the Bay Area.
No horizon
Only if: when nobody in Washington knows where they will be come January, and pots of gold await in the Bay Area
Source: We can’t trust the fox to guard the henhouse, especially when it comes to AI
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“We waited too long.”
Our reading: We waited too long to act against AI-generated misinformation and deepfakes in elections.
No horizon
Source: We can’t trust the fox to guard the henhouse, especially when it comes to AI
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Instead we need to wake up and realize that hallucinations are absolutely core to how LLMs work, and that we need new approaches based on different ideas.”
Our reading: The AI field needs new approaches based on ideas other than LLMs to address hallucinations.
No horizon
Qualifier: because hallucinations are core to LLMs
Source: Facing Facts
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“TL;DR: Don't console yourself with making something that superficially looks like human errors, if you aspire to AGI.”
Our reading: Those aiming for AGI should not take comfort in LLM errors superficially resembling human errors.
No horizon
Qualifier: if you aspire to AGI
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Rather, the goal of AGI should be to build machines that can reliably reason and plan about a wide swathe of the world.”
Our reading: The goal of AGI should be machines that reliably reason and plan about a wide swathe of the world, not recreating humans.
No horizon
Assessment: Can’t be checkedSaid
Forecast
“The field of AI will eventually do better, but probably with an AI that is structured differently, in which facts are first-class citizens, rather than something you hope you might get for free with enough data.”
Our reading: AI will eventually do better on hallucinations, probably via differently structured systems that treat facts as first-class rather than via more data.
No horizon
Qualifier: eventually
Assessment: Can’t be checkedSaid
Causal claim
“But even with perfect data, the stochastic reconstructive process would still produce some errors.”
Our reading: Even with perfect data, the stochastic reconstructive process LLMs use would still produce some errors.
No horizon
Qualifier: even with perfect data
Assessment: Can’t be checkedSaid
Conditional forecast
“Gary Marcus is shaking his head, but cautiously optimistic that an increase of realism might lead to a renaissance of new approaches to AI.”
Our reading: An increase in realism about AI might lead to a renaissance of new approaches to AI.
No horizon
Only if: if realism about AI increases
Source: Groupthink versus Tulips. What will historians think of 2024?
Assessment: Can’t be checkedSaid
Observation
“Meanwhile, and perhaps not coincidentally, the bubble deflation that I have been warning since August of has finally perhaps begun:”
Our reading: The AI bubble deflation may have begun as of April 2024.
No horizon
Qualifier: perhaps
Source: Groupthink versus Tulips. What will historians think of 2024?
Assessment: Can’t be checkedSaid
Forecast
“So 2024 could be remembered as the high point in a kind of tulip-mania.”
Our reading: 2024 could be remembered as the peak of an AI tulip-mania.
No horizon
Source: Groupthink versus Tulips. What will historians think of 2024?
Assessment: Can’t be checkedSaid
Rhetoric
“Historians are going to be scratching their heads.”
Our reading: Historians will be puzzled by the AI industry's groupthink.
No horizon
Source: Groupthink versus Tulips. What will historians think of 2024?
Assessment: Can’t be checkedSaid
Forecast
“Strikingly, Meta is giving it away free, and the more deeply-trained version (still to come/still being trained) may well be in the general vicinity of GPT-4.”
Our reading: The more deeply-trained version of Meta's LLAMA-3, still being trained, may well be in the general vicinity of GPT-4.
No horizon
Scope: more deeply-trained version still to come/still being trained
Source: Groupthink versus Tulips. What will historians think of 2024?
Assessment: Can’t be checkedSaid
Causal claim
“Dreamers who promise far more than they are likely to deliver are getting almost all the money, and (at least in AI) they are getting so much money for a path that it is unlikely to work that they are leaving too little oxygen for the exploration of new ideas that might work.”
Our reading: Dreamers who promise far more than they are likely to deliver are getting almost all the money, and, at least in AI, they are getting so much money for a path unlikely to work that too little oxygen is left for exploring new ideas that might work.
No horizon
Scope: at least in AI (applies to the crowding-out claim)
Assessment: Can’t be checkedSaid
Forecast
“Dreamers who promise far more than they are likely to deliver are getting almost all the money, and (at least in AI) they are getting so much money for a path that it is unlikely to work that they are leaving too little oxygen for the exploration of new ideas that might work.”
Our reading: The dominant, heavily funded current AI approach is unlikely to work.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“There is also a hostility in some parts of Silicon Valley right now to regulation. This a mistake.”
Our reading: The hostility to regulation in some parts of Silicon Valley right now is a mistake.
No horizon
Scope: some parts of Silicon Valley
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Right now I think the current net effect of venture capital is not great.”
Our reading: Marcus thinks that right now the net effect of venture capital is not great.
No horizon
Qualifier: I think
Assessment: Can’t be checkedSaid
Conditional forecast
“I don’t see how we get to a positive AI future if we sweep all the problems under the rug.”
Our reading: Marcus does not see how a positive AI future can be reached if all of AI's problems are swept under the rug.
No horizon
Only if: if we sweep all the problems under the rug
Assessment: Can’t be checkedSaid
Conditional forecast
“But we can only get there if we can face the limitations and risks honestly.”
Our reading: Technologies that improve the world can only be reached if we can face their limitations and risks honestly.
No horizon
Only if: unless we can face the limitations and risks honestly
Assessment: Can’t be checkedSaid
Forecast
“I doubt that anyone with a short leash can achieve the breakthroughs we will need, if we are to get to the optimistic future so many of us would like to see.”
Our reading: AI efforts funded on a short leash are unlikely to achieve the breakthroughs needed for an optimistic AI future.
No horizon
Qualifier: with a short funding leash
Assessment: Can’t be checkedSaid
Forecast
“I think Khosla’s timing (all this by 2049) might be off, perhaps by decades, but I don’t see why we can’t get to most of this eventually.”
Our reading: Khosla's predicted technological advances may not all arrive by 2049, perhaps being decades late, though most could be achieved eventually.
No horizon
Qualifier: perhaps off by decades
Assessment: Can’t be checkedSaid
Observation
“I don’t think we are any closer to a machine with the wherewithal or the motivation to annihilate the human species than we were a year ago.”
Our reading: He does not think we are any closer to a machine with the wherewithal or the motivation to annihilate the human species than we were a year ago.
No horizon
Qualifier: I don't think
Source: TED, AI, and the test of time
Assessment: Can’t be checkedSaid
Observation
“I love Khan Academy, but continue to have doubts about whether LLMs are really the right tool for the next step of the job.”
Our reading: He continues to have doubts about whether LLMs are really the right tool for the next step of the job of transforming education.
No horizon
Qualifier: continues to have doubts
Source: TED, AI, and the test of time
Assessment: Can’t be checkedSaid
Rhetoric
“Both damning and correct. People who believe in LLMs have played themselves.”
Our reading: Bjarnason's LLMentalist analysis is correct, and people who believe in LLMs have fooled themselves.
No horizon
Source: LLMs aren’t very bright. Why are so many people fooled?
Assessment: Can’t be checkedSaid
Causal claim
“Gary Marcus thinks the whole world would be different if so many people weren’t so easily taken by the parlor tricks of generative AI.”
Our reading: Gary Marcus thinks the whole world would be different if so many people were not so easily taken in by the parlor tricks of generative AI.
No horizon
Qualifier: Gary Marcus thinks
Source: LLMs aren’t very bright. Why are so many people fooled?
Assessment: Can’t be checkedSaid
Observation
“The second summary, deeply parallel with the first, outlines the process by which an unthinking LLM (without meaning to do so) cons many of its users.”
Our reading: An unthinking LLM, without meaning to do so, cons many of its users through a process that closely parallels a psychic's con.
No horizon Evidence as of
Limitation: The essay page shows no publication date. Its date is recorded as 2023 based on Marcus's note ('from last year'), and its available_at is the post date.
Source: LLMs aren’t very bright. Why are so many people fooled?
Assessment: Can’t be checkedSaid
Forecast
“Reliable, trustworthy AI is surely achievable, but we may need to go back to the drawing board to get there.”
Our reading: Reliable, trustworthy AI is surely achievable, but we may need to go back to the drawing board to get there.
No horizon
Qualifier: achievability stated as 'surely'
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: Can’t be checkedSaid
Forecast
“In the most extreme case, OpenAI’s $86 billion valuation could look in hindsight like a WeWork moment for AI.”
Our reading: In the most extreme case, OpenAI's $86 billion valuation could in hindsight look like a WeWork moment for AI.
No horizon
Qualifier: in the most extreme case
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: Can’t be checkedSaid
Observation
“What really got me to thinking about all this was a graph from OpenAI a couple days ago, touting their latest model, GPT 4 Turbo, which I have always suspected was a failed attempt at GPT-5.”
Our reading: Marcus suspects GPT-4 Turbo was a failed attempt at building GPT-5.
No horizon
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: Can’t be checkedSaid
Observation
“My gut sense is that we haven’t reached the real ceiling on MMLU yet, and that this is a genuine sign of diminishing returns.”
Our reading: The MMLU plateau is likely a genuine sign of diminishing returns rather than a ceiling effect.
No horizon
Qualifier: MMLU real ceiling not yet reached
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: Can’t be checkedSaid
Forecast
“The wall that I once warned about, in 2022, may finally be approaching.”
Our reading: LLM progress may be approaching the wall Marcus warned about in 2022.
No horizon
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: Can’t be checkedSaid
Conditional forecast
“If we really have changed regimes, from rapid progress to diminishing returns, and hallucinations and stupid errors do linger, LLMs may never be ready for prime time.”
Our reading: If we really have changed regimes from rapid progress to diminishing returns, and hallucinations and stupid errors do linger, LLMs may never be ready for prime time.
No horizon
Only if: if we really have changed regimes from rapid progress to diminishing returns
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: Can’t be checkedSaid
Forecast
“But that place may be smaller; it is entirely possible that LLMs on their own will never live up to last year’s wild expectations.”
Our reading: LLMs' role may be smaller than hoped, and LLMs alone may never live up to 2023's wild expectations.
No horizon
Qualifier: LLMs on their own
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: Can’t be checkedSaid
Observation
“I could be persuaded that on some measures there was a doubling of capabilities for some set of months in 2020-2023, but I don’t see that case at all for the last 13 months.”
Our reading: Marcus could be persuaded that LLM capabilities doubled on some measures for some months in 2020-2023, but he does not see that case at all for the 13 months since GPT-4's release.
No horizon Evidence as of
Limitation: The packet contains no evidence items establishing a defined capability comparison.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: Can’t be checkedSaid
Observation
“Whatever doubling there might have been, has perhaps come to an end.”
Our reading: Any doubling of LLM capabilities has perhaps come to an end.
No horizon
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: Can’t be checkedSaid
Observation
“If two graphs I plotted are remotely correct, Mollick’s claim that “the rate of improvement in Large Language models show capabilities doubling every 5 to 14 months” is no longer holding true.”
Our reading: If his graphs are roughly correct, LLM capabilities are no longer doubling every 5 to 14 months.
No horizon Evidence as of
Limitation: This does not endorse Mollick's wording.
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: Can’t be checkedSaid
Forecast
“Instead, as I warned in August, we may well be in for a correction.”
Our reading: The AI market may well be in for a correction.
No horizon
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: Can’t be checkedSaid
Observation
“And of course advancing on benchmarks isn’t itself enough anyway; few benchmarks capture the complexity of the real world. Even if LLMs could max out on all existing benchmark, we might still have a long way to go.”
Our reading: Advancing on benchmarks is not enough by itself because few benchmarks capture the complexity of the real world, and even if LLMs could max out all existing benchmarks, we might still have a long way to go.
No horizon
Qualifier: even if LLMs could max out on all existing benchmarks
Source: Evidence that LLMs are reaching a point of diminishing returns - and what that might mean
Assessment: Can’t be checkedSaid
Forecast
“Learn and master the basics of almost any new video game within a few minutes or hours, and solve original puzzles in the alternate world of that video game.”
Our reading: Marcus seriously doubts that, by the end of 2025, AI will be able to learn and master the basics of almost any new video game within a few minutes or hours and solve original puzzles in the alternate world of that video game.
Deadline passed Evidence as of
Limitation: This record does not show what AI systems could do with new video games in 2025. No evidence on that question is part of this record.
Source: Superhuman AGI is not nigh
Assessment: Can’t be checkedSaid
Forecast
“Write cogent, persuasive legal briefs without hallucinating any cases”
Our reading: AI is unlikely by end of 2025 to write cogent, persuasive legal briefs without hallucinating any cases.
No horizon
Source: Superhuman AGI is not nigh
Assessment: Can’t be checkedSaid
Forecast
“Watch previously unseen mainstream movie and be able to follow plot twists and know when to laugh, and be able to summarize it without giving away any spoilers or making up anything that didn’t actually happen”
Our reading: AI is unlikely by end of 2025 to watch an unseen mainstream movie, follow plot twists, know when to laugh, and summarize it without spoilers or fabrications.
No horizon
Source: Superhuman AGI is not nigh
Assessment: Can’t be checkedSaid
Forecast
“None seem to me to be within immediate reach.”
Our reading: None of the listed tasks seem within immediate reach of AI.
No horizon
Source: Superhuman AGI is not nigh
Assessment: Can’t be checkedSaid
Observation
“Right now there are likely hundreds, maybe thousands or even millions of such things.”
Our reading: As of April 2024 there are likely hundreds or more pieces of intellectual work that the best humans can do and AI cannot.
No horizon
Source: Superhuman AGI is not nigh
Assessment: Can’t be checkedSaid
Forecast
“Walk into random kitchens and prepare a cup of coffee or make breakfast”
Our reading: AI is unlikely by end of 2025 to walk into random kitchens and prepare coffee or breakfast.
No horizon
Source: Superhuman AGI is not nigh
Assessment: Can’t be checkedSaid
Forecast
“Babysit children in an unfamiliar homes and keep them safe”
Our reading: AI is unlikely by end of 2025 to babysit children in unfamiliar homes and keep them safe.
No horizon
Source: Superhuman AGI is not nigh
Assessment: Can’t be checkedSaid
Forecast
“Drive off-road vehicles, without maps, across streams, around obstacles such as fallen trees, and so on.”
Our reading: AI is unlikely by end of 2025 to drive off-road vehicles without maps across streams and around obstacles.
No horizon
Source: Superhuman AGI is not nigh
Assessment: Can’t be checkedSaid
Forecast
“Clean unfamiliar houses and apartments”
Our reading: AI is unlikely by end of 2025 to clean unfamiliar houses and apartments.
No horizon
Source: Superhuman AGI is not nigh
Assessment: Can’t be checkedSaid
Forecast
“Learn to ride a mountain bike off-road through forest trails”
Our reading: AI is unlikely by end of 2025 to learn to ride a mountain bike off-road through forest trails.
No horizon
Source: Superhuman AGI is not nigh
Assessment: Can’t be checkedSaid
Forecast
“Tend to the physical and psychological needs of an elderly or infirm person”
Our reading: AI is unlikely by end of 2025 to tend to the physical and psychological needs of an elderly or infirm person.
No horizon
Source: Superhuman AGI is not nigh
Assessment: Can’t be checkedSaid
Forecast
“Reliably drive a car in a novel location that they haven’t previously encountered, even in the face of unusual circumstances like hand-lettered signs, without the assistance of other humans.”
Our reading: AI is unlikely by end of 2025 to reliably drive a car in novel locations with unusual circumstances without human assistance.
No horizon
Qualifier: without assistance of other humans
Source: Superhuman AGI is not nigh
Assessment: Can’t be checkedSaid
Observation
“In another bit of possible foreshadowing: GenAI is starting to get a tiny bit of Scarlet Letter feel.”
Our reading: Generative AI is starting to get a tiny bit of a 'Scarlet Letter' feel, which he presents as possible foreshadowing.
No horizon
Qualifier: a tiny bit
Source: 24 hours in AI news
Assessment: Can’t be checkedSaid
Forecast
“Another sign that the bubble may not last forever.”
Our reading: The generative AI investment bubble may not last forever, as signalled by some late-stage VC firms sitting out.
No horizon
Qualifier: based on some late-stage VCs sitting out the GenAI rush
Source: 24 hours in AI news
Assessment: Can’t be checkedSaid
Observation
“Not one of these problems has been adequately addressed.”
Our reading: None of the problems listed (hallucinations, abstraction and reasoning, brittleness, data guzzling, etc.) has been adequately addressed.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“We must invest heavily in alternative approaches and stop funding a losing hand to the infinite degree, crowding everything out.”
Our reading: The field must invest heavily in alternatives to generative AI and stop pouring unlimited funding into it, which crowds out everything else.
No horizon