“But the truth is it doesn’t work that well yet.”
Our reading, the sentence that was checked:
Programming in English with LLMs does not yet work very well.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, The hottest new programming language is English! Or maybe not., paragraph 5.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-the-hottest-new-programming-language-20535b-a2
We recorded 4 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-02-15. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- Programming in English
- With LLMs
- doesn’t work that well yet
Our interpretation
- The plain reading concerns dependable completion of English programming requests, not whether LLMs have any programming ability or usefulness.
- Support requires substantial correctness or completion failures on clearly specified programming tasks. An occasional error alone would not establish the claim.
- Because no reporting period is specified, assess early 2024: the latest completed calendar month and the elapsed portion of February before publication.
Counts as supported if: Contemporaneous evaluations show substantial failures to produce complete, correct programs from clear natural-language specifications, with testing or repair still necessary.
Counts as contradicted if: Broad evaluations of contemporaneous systems show dependable completion of varied programming requests without substantial correction, making the stated reliability concern unrepresentative.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The study published January 16, 2024 reported GPT-4 pass@5 accuracy on CodeContests validation problems of 19% with a single-prompt method and 44% with AlphaCodium’s testing and revision workflow [e1]. AlphaCodium generated, ran and repaired code against public and AI-generated tests [e1]. These results support the concern about dependable correctness on the tested competitive-programming tasks, while also showing useful capability and improvement with testing and repair [e1].
Limitations
- The competitive-programming evaluation does not establish a representative success rate for everyday programming or direct control through English prompts [e1].
- The assessment interprets 'works well' as dependable correctness; the evidence supplies no numerical threshold for satisfactory performance.
- The December 2023 code-editing results fall outside the selected January 1–February 15, 2024 assessment period and provide context only [e2].
Evidence (2 items)
Supports the claim: arXiv; Tal Ridnik, Dedy Kredo and Itamar Friedman,
accuracy improved from 19% to 44%.
Why it matters here: On CodeContests validation problems, direct prompting solved 19% with five attempts; a structured testing and revision process reached 44%. This supports substantial reliability limitations while also documenting useful capability and improvement.
- Publisher
- arXiv; Tal Ridnik, Dedy Kredo and Itamar Friedman
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: Aider; Paul Gauthier,
lazy comments
Why it matters here: The earlier refactoring benchmark documented missing implementations replaced by comments. Its deliberately challenging task selection and December timing prevent treating it as a representative early-2024 failure rate.
- Publisher
- Aider; Paul Gauthier
- Kind of source
- Independent measurement
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, The hottest new programming language is English! Or maybe not.
But the truth is it doesn’t work that well yet.
- Where
- paragraph 5
- Published
- 2024-02-15
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-the-hottest-new-programming-language-20535b-a2: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-the-hottest-new-programming-language-20535b-a2.