“Reliable agents may not require “artificial general intelligence”, but they surely require what I called in 2020 “robust”, trustworthy AI. LLMs aren’t getting us there.”
Our reading, the sentence that was checked:
Reliable agents require robust, trustworthy AI, and LLMs are not getting us there.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, AI Agents have, so far, mostly been a dud, paragraph 45.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-ai-agents-have-so-far-mostly-been-ab9cd9-a3
We recorded 23 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-08-03. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Reliable agents may not require “artificial general intelligence”
- they surely require
- “robust”, trustworthy AI
- LLMs aren’t getting us there
Our interpretation
- The ordinary reading of the final sentence concerns the direction of progress toward robust, trustworthy agents. It is stronger than saying agents had not yet achieved reliability.
- Robust, trustworthy agents are read as agents that perform ordinary delegated tasks dependably, including under foreseeable variations and disruptions. This operational description is the evaluator's interpretation.
- The necessity statement about robustness identifies the destination. It does not by itself establish the empirical assertion about LLM progress.
- Because the statement describes an ongoing state without a reporting period, July 2025, the latest completed month before the post, is the assessment period. Earlier measurements can supply historical context for that trajectory.
- The surrounding doubts about what LLMs will ever achieve do not turn this present-tense claim into a forecast with an invented deadline.
Counts as supported if: Comparable evidence about the trajectory into July 2025 shows that successive LLM-based agents were not making meaningful, transferable progress toward dependable task completion and resilience to foreseeable disruptions. Merely documenting continued failures is insufficient.
Counts as contradicted if: Comparable evidence about the trajectory into July 2025 shows meaningful, transferable improvements in dependable task completion and resilience across ordinary agent tasks. Reaching perfect reliability is unnecessary, but isolated capability gains do not alone establish this broader progress.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The claim concerns progress toward trustworthy agents, not simply whether agents were already trustworthy. METR supplies material evidence of increasing task horizons. The user-coordination benchmark supplies evidence of continuing weaknesses, but those weaknesses do not establish a stalled trajectory. A later reliability study finds small improvements, yet its aggregate result does not settle the July 2025 observation. The reviewed evidence therefore establishes neither the broad lack of progress asserted nor a sufficiently broad contrary trajectory.
Limitations
- The reporting window and operational meaning of progress are evaluator interpretations.
- Software task horizons, user coordination, and broad operational reliability measure different aspects of agent performance.
- The available evidence does not justify treating benchmark gains as complete proof of trustworthy operation or persistent failures as proof of no progress.
Evidence (4 items)
Context only: METR researchers via arXiv,
Furthermore, frontier AI time horizon has been doubling approximately every seven months since 2019, though the trend may have accelerated in 2024.
Why it matters here: This is material counterevidence to a claim of no agent progress: the study measures longer tasks completed at a fixed success probability. Its March findings and software-focused task suite do not establish the broader July trajectory in trustworthy operation.
- Publisher
- METR researchers via arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: τ²-bench researchers via arXiv,
In particular, our experiments show significant performance drops when agents shift from no-user to dual-control, highlighting the challenges of guiding users.
Why it matters here: The experiments identify a relevant weakness in agents that must guide users. This comparison between operating conditions does not show whether successive LLM systems were advancing toward reliability.
- Publisher
- τ²-bench researchers via arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: Mahmoud Mohammadi and coauthors via arXiv,
The rise of LLM-based agents has opened new frontiers in AI applications, yet evaluating these agents remains a complex and underdeveloped area.
Why it matters here: This contemporaneous survey identifies limitations in evaluating agents. It helps explain the measurement gap but supplies no outcome test of Marcus's trajectory claim.
- Publisher
- Mahmoud Mohammadi and coauthors via arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: Stephan Rabanser and coauthors via arXiv,
Evaluating 14 agentic models across two complementary benchmarks, we find that recent capability gains have only yielded small improvements in reliability.
Why it matters here: This later evaluation distinguishes capability gains from reliability gains and reports limited improvement. Its aggregate result concerns a later evaluation and cannot establish the trajectory specifically into July 2025.
- Publisher
- Stephan Rabanser and coauthors via arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
Comparable evaluations of successive LLM-based agents covering the trajectory into July 2025, with repeated ordinary tasks and foreseeable disruptions, could establish whether reliability improvements transferred across settings or consistently failed to do so.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, AI Agents have, so far, mostly been a dud
Reliable agents may not require “artificial general intelligence”, but they surely require what I called in 2020 “robust”, trustworthy AI. LLMs aren’t getting us there.
- Where
- paragraph 45
- Published
- 2025-08-03
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-ai-agents-have-so-far-mostly-been-ab9cd9-a3: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-ai-agents-have-so-far-mostly-been-ab9cd9-a3.
HeldBy this site
The anthropic review found the criteria unfair to the speaker's words.
About assessment marcus-ai-agents-have-so-far-mostly-been-ab9cd9-a2, which was not published.