AI Coding Fantasy meets Pac-Man
We recorded 9 claims from this post. 7 were checked against evidence. 2 can’t be checked. 4 more are held. Read the post at its source.
Claims from this post (9)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
Observation
“A few days ago he triumphantly claimed (without evidence) that nearly all code would be written by AI within the next year:”
Our reading: Dario Amodei's claim that nearly all code would be written by AI within the next year was made without evidence.
No horizon Evidence as of
Limitation: The readable transcription is unofficial.
Assessment: Can’t be checkedSaid
Forecast
“I think most or all of the points I made – about challenges in building and debugging novel systems with sufficient generality- will stand the test of time.”
Our reading: Marcus thinks that most or all of the points he made last week, about the challenges of building and debugging novel systems with sufficient generality, will stand the test of time.
No horizon
Qualifier: I think
Assessment: SupportedSaid
Observation
“I don’t want spoil all the fun, but … things didn’t go well:”
Our reading: In The Guardian's test, people using AI tools without coding knowledge did not succeed well at building a Pac-Man clone.
No horizon Evidence as of
Limitation: The report does not establish universal failure or a population failure rate [e1, e2].
Assessment: SupportedSaid
Forecast
“To be sure, most or all trained coders will (if they are not already) soon use AI as a part of their workflow.”
Our reading: Most or all trained coders will soon use AI as part of their workflow, if they are not already.
Settled before its deadline Evidence as of
Limitation: These are self-reported surveys, not a census.
Assessment: UnresolvedSaid
Forecast
“AI will be a tool to help people write code, just as spell-checkers are a tool to help authors write articles and novel, but AI will not soon replace people who understand how to conceive of, write, and debug code.”
Our reading: AI will serve as a tool to help people write code but will not soon replace people who know how to conceive of, write, and debug code.
No deadline stated Evidence as of
Limitation: The available employment research does not directly measure responsibility for conceiving and debugging software.
Assessment: MixedSaid
Causal claim
“As a fantastic, souped-up version of autocomplete, AI coding tools somewhat increase productivity.”
Our reading: AI coding tools, acting as a fantastic, souped-up version of autocomplete, somewhat increase productivity.
No horizon Evidence as of
Limitation: Neither study establishes an industry-wide average effect or isolates autocomplete from other assistance features [e1, e2].
Assessment: UnresolvedSaid
Observation
“And they don’t magically resolve the challenges of maintaining complex code bases over time.”
Our reading: Current AI coding tools do not resolve the challenges of maintaining complex code bases over time.
No horizon Evidence as of
Limitation: The study does not establish whether its findings apply to target-period tools [e1].
Assessment: UnresolvedSaid
Forecast
“The idea that coders (and more generally, software architects) are on their way out is absurd.”
Our reading: Coders and software architects are not on their way out due to AI.
No deadline stated Evidence as of
Limitation: The ordinary occupational-obsolescence reading differs from a claim that employment will never decline.
Assessment: Can’t be checkedSaid
Observation
“My guess is that current AI coding tools would struggle to use 6502 well, for multiple reasons, including a lack of high-level libraries, fewer Stack Overflow queries to rely on, and a greater needed for the coding tool to understanding high level architectural concerns that might be side-stepped in higher level languages.”
Our reading: Marcus guesses that current AI coding tools would struggle to use 6502 assembly well, for multiple reasons including a lack of high-level libraries, fewer Stack Overflow queries to rely on, and a greater need to understand high-level architectural concerns that might be side-stepped in higher-level languages.
No horizon
Qualifier: my guess
Held from this post (4)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The record does not meet the current rules. Our reading, not assessed: In some instances, beginners' overdependence on AI coding tools will produce large amounts of hard-to-maintain spaghetti code and create security problems.
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: AI coding tools can help a coder learn a new API, and maybe even a new programming language.
- Attribution is uncertain