“This is why so called “code agents” inevitably fall apart as soon as one gives them a more complex task which requires reasoning.”
Our reading, the sentence that was checked:
Code agents inevitably fail on more complex tasks requiring reasoning.
Assessment: Contradicted Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI, paragraph 117.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-how-o3-and-grok-4-accidentally-vindicated-e81e95-a1
We recorded 47 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2025-07-13. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- given a more complex task requiring reasoning (qualifier, stated by the speaker)
From the speaker
- code agents
- inevitably fall apart
- as soon as one gives them a more complex task
- requires reasoning
Our interpretation
- The plain reading of inevitably is unavoidable failure, rather than frequent failure or inadequate reliability.
- Complex reasoning tasks include multistep repository diagnosis and repair, beyond generating a routine isolated snippet.
- I assess contemporary agents during 2025 through the post date, using the current year-to-date benchmark reporting window.
- The outcome assessment addresses the asserted failure pattern. It does not separately establish the preceding explanation about tool selection and integration.
Counts as supported if: Broad evaluations establish failure across the relevant class of complex reasoning tasks, with no credible reproducible successes.
Counts as contradicted if: Documented, reproducible evaluations show contemporary code agents successfully completing complex multistep software tasks.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Contradicted
The dated evidence goes against what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
Contemporary agents demonstrably completed multistep repository repairs, and the released evaluation permits those successes to be examined. That contradicts inevitable failure on complex tasks under the ordinary reading. Marcus's weaker concern remains credible: reliability declines substantially as tasks become longer and harder. Successful cases do not establish dependable autonomous software engineering.
Limitations
- The result depends on taking inevitably literally and treating repository diagnosis and repair as complex reasoning.
- Passing benchmark tests does not establish complete production readiness.
- The METR citation uses the original article's prose, not its subsequently updated interactive chart.
- Experiment execution dates are not specified separately from the dated reports.
Evidence (2 items)
Goes against the claim: Yang et al., arXiv,
40.2% Pass@1 resolve rate
Why it matters here: The documented agent edits files and executes commands to resolve real repository issues, with released methods and assets. The accompanying analysis reports multistep successful trajectories and nonzero success even on tasks rated hard, contradicting unavoidable failure under the stated reading.
- Publisher
- Yang et al., arXiv
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: METR,
unable to reliably handle
Why it matters here: METR reports severe reliability limits, including success below 10% on tasks taking humans more than about four hours, while also reporting some successes on tasks taking experts hours. This supports Marcus's broader concern about reliability, but not inevitable failure.
- Publisher
- METR
- Kind of source
- Independent measurement
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, How o3 and Grok 4 Accidentally Vindicated Neurosymbolic AI
This is why so called “code agents” inevitably fall apart as soon as one gives them a more complex task which requires reasoning.
- Where
- paragraph 117
- Published
- 2025-07-13
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-how-o3-and-grok-4-accidentally-vindicated-e81e95-a1: Current. Outcome: Contradicted. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-how-o3-and-grok-4-accidentally-vindicated-e81e95-a1.