“(Amateurs can in fact now build prototypes, but the auto-generated code tends to be shaky and few amateurs can make those prototypes robust.)”
Our reading:
Amateurs can build prototypes with AI coding, but the generated code tends to be shaky and few can make it robust.
Assessment: Unresolved
What happened
Was it true for the period described?
Outcome: Unresolved
We checked, and the evidence so far doesn’t settle it.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
The hackathon study, published December 2, 2025, reports functional demonstrations and uneven code quality among novice and mixed-experience teams. Its event is dated only to 2025, and it includes no production-hardening follow-up [e1]. The other study used March–April 2025 videos and describes debugging that combines AI assistance with manual work [e2]. Neither establishes how commonly amateurs encountered material defects or successfully hardened prototypes during November 1–29, 2025, leaving the prevalence claims unresolved [e1, e2].
Caveats
- Neither study provides representative success rates for amateurs hardening prototypes.
- The hackathon mixes experience levels and measures demonstrations rather than sustained dependability.
- The studies do not jointly measure correctness, security and operational reliability after sustained development.
Evidence (2 items)
Context only: Kiev Gama and colleagues, arXiv,
uneven code quality requiring rework
Why it matters: The study reports functional demonstrations and quality problems in a short hackathon involving novice and mixed-experience teams. The event date is not specified more precisely, and the design includes no production-hardening follow-up.
- Where
- Abstract
- Publisher
- Kiev Gama and colleagues, arXiv
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
Context only: Advait Sarkar and Ian Drosos, arXiv,
AI assistance with manual practices.
Why it matters: The qualitative study documents continued human involvement in debugging. Its video sample was collected in March–April and does not measure how often amateurs could harden applications in November.
- Publisher
- Advait Sarkar and Ian Drosos, arXiv
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Conditions
- for amateurs (their words)
Where we had to interpret
- Amateurs means people without professional software-development expertise, including non-programmers and beginners.
- A prototype demonstrates intended functionality. Robust means sufficiently correct, secure and dependable for its intended real use, including foreseeable errors and edge cases.
- Tends to be shaky means quality problems are common, rather than merely possible. Few means successful hardening is uncommon among amateurs attempting it; no exact percentage is imposed.
- Use November 1–29, 2025, the latest month-to-publication window for the ongoing observation. Include developments through publication rather than stopping in September.
- Earlier studies and later publications can inform the assessment, but only observations tied to the target period can decide its contemporary prevalence claims.
Right if: Studies show amateurs producing functional prototypes, commonly encountering material generated-code defects, and only a small share successfully making those prototypes dependable for their intended use.
Wrong if: Representative evaluations show amateurs routinely producing dependable applications or successfully hardening prototypes, with material generated-code problems uncommon.
We wrote this test after the outcome was known.
What would settle it
A representative study tied to November 1–29, 2025, identifying participants’ expertise and tool versions, independently testing prototype defects, and reporting how often amateurs achieved intended-use correctness, security and reliability after a reasonable opportunity to harden their applications.
We found 34 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be, paragraph 64.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-28. The post may have been edited after 2025-11-29.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, Three years on, ChatGPT still isn't what it was cracked up to be – and it probably never will be
(Amateurs can in fact now build prototypes, but the auto-generated code tends to be shaky and few amateurs can make those prototypes robust.)
- Where
- paragraph 64
- Published
- 2025-11-29
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Unresolved. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
A reviewer said our test wasn’t fair to what was actually said.