No, Virginia, AGI is not imminent
We recorded 17 claims from this post. 9 were checked against evidence. 8 can’t be checked. 3 more are held. Read the post at its source.
Claims from this post (17)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Forecast
“I seriously doubt it.”
Our reading: Marcus seriously doubts that AI is close to beating typical human performance on virtually all cognitive tasks.
No horizon
Qualifier: AGI as defined by Legg
Assessment: UnresolvedSaid
Observation
“in my view it’s a subtle redefinition that dramatically moves the goal posts.”
Our reading: In Marcus's view, Legg's definition of AGI as better than typical human performance on virtually all cognitive tasks is a subtle redefinition that dramatically moves the goal posts, compared with how Legg and Marcus publicly discussed the term last year.
No horizon Evidence as of
Limitation: Missing original exchange records prevent a finding either way.
Assessment: SupportedSaid
Observation
“But it is also well-known throughout the industry that benchmarks are often quite gameable.”
Our reading: AI benchmarks are often quite gameable.
No horizon Evidence as of
Limitation: The evidence does not estimate the proportion of all benchmarks affected or measure industry-wide awareness [e1, e2].
Assessment: SupportedSaid
Observation
“Basically, beating benchmarks doesn’t tell you as much as you might think. Often when you do, it’s a statistical artifact.”
Our reading: Beating AI benchmarks is often a statistical artifact and says less than people assume.
No horizon Evidence as of
Limitation: No representative estimate establishes how often this happens across all AI benchmarks.
Assessment: UnresolvedSaid
Causal claim
“This is even more true four years later, now that we have systems that ingest massive amounts of data from both public and commercial sources, quite possibly including eg sample law school and medical exams etc.”
Our reading: Because current systems ingest massive data possibly including sample exams, benchmark success is even less indicative of intelligence than in 2019.
No horizon Evidence as of
Limitation: OpenAI's audit was a developer assessment with acknowledged detection limits.
Assessment: SupportedSaid
Observation
“The dirty little secret that pretty much everyone in the AI community knows is that doing well on a benchmark is not actually tantamount to being intelligent.”
Our reading: Doing well on a benchmark does not amount to being intelligent.
No horizon Evidence as of
Limitation: The assessment depends on reading intelligent as broad, reliable competence in the surrounding AGI discussion.
Assessment: UnresolvedSaid
Observation
“Without a sense of what it is in the training corpus, it’s almost impossible to conclude anything about what a machine truly understands and how general that is.”
Our reading: Without knowing the training corpus, one can hardly conclude what a machine understands or how general it is.
No horizon Evidence as of
Limitation: A reading limited to whether any bounded behavioral conclusion is possible would weigh more strongly against Marcus.
Assessment: Can’t be checkedSaid
Observation
“Humans can write summaries and reliably answer many questions about original films and television programs that they watch; I doubt LLMs (or any other extant AI technology) can do this at all.”
Our reading: Marcus doubts LLMs or any extant AI can summarize and reliably answer questions about original films and TV programs.
No horizon
Assessment: UnresolvedSaid
Observation
“Humans can write summaries of things without hallucination; LLMs cannot do so reliably.”
Our reading: LLMs cannot reliably write summaries without hallucinating.
No horizon Evidence as of
Limitation: The inferred monthly observation window is narrower than a general review of 2023 models.
Assessment: Can’t be checkedSaid
Observation
“And frankly, we still have a long way to go on all the things I always harp on: reasoning, compositionality, planning, factuality, and so on.”
Our reading: AI still has a long way to go on reasoning, compositionality, planning, and factuality.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“No AGI worth its salt would brick multidigit arithmetic.”
Our reading: A genuine AGI must reliably do multidigit arithmetic.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“It’s a huge downgrade in our goals for artificial general intelligence to say “better than the average untrained human on many tasks”; the point is that we should expect machines to do what we ask them to do;”
Our reading: Saying AGI means 'better than the average untrained human on many tasks' is a huge downgrade in our goals for AGI, because we should expect machines to do what we ask them to do.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“An artificial general intelligence should do the things that it is asked to do, and decline to do those that it can’t, and have the wisdom to know the difference.”
Our reading: An AGI should do what it is asked, decline what it cannot do, and know the difference.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Dropping from “as smart as the Star Trek computer” to better than Joe Sixpack should not be our goal.”
Our reading: The AI field should not lower its AGI goal from Star Trek computer-level to merely beating an average person.
No horizon
Assessment: UnresolvedSaid
Observation
“18 months ago, before the LLM euphoria, we all took for granted that AGI wasn’t just about beating ordinary humans.”
Our reading: Around early 2022, before LLM euphoria, the field assumed AGI meant more than beating ordinary humans.
No horizon Evidence as of
Limitation: The later review provides historical definitions, not a representative measurement of February 2022 views [e1].
Assessment: SupportedSaid
Observation
“What would make artificial general intelligence general is generality: the ability to cope with new things. And that’s precisely where current AI is still lacking.”
Our reading: Generality, the ability to cope with new things, is what would make AGI general, and that is precisely where current AI is still lacking.
No horizon Evidence as of
Limitation: The outcome depends on reading 'lacking' as substantial limitations, rather than complete absence.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Let’s not dumb down our standards, and wind up with a bunch of machines we can’t rely on, just so that big tech can prematurely declare victory, when there is plenty of work left to do.”
Our reading: AGI standards should not be lowered so big tech can prematurely declare victory, since much work remains.
No horizon
Held from this post (3)
These records did not pass every check, so nothing about them is published. Held is not a finding.