Unlike humans, generative AI does not know what a coding mistake looks like, because as a probabilistic model it doesn't know what anything is.

Not graded on evidence

Outcome: Can’t be checked

An opinion, a recommendation, or too vague to test. Listed, not scored.

Evidence up to 2026-10-28. Checked 2026-10-31.

Why

He says generative AI does not know what a coding mistake looks like, or what anything is, because it is a probabilistic model. This is a claim about knowledge and understanding, not about measurable performance. The record has two related studies. A June 2024 preprint by OpenAI authors (e1) trained critic models with RLHF to review code. On code with naturally occurring LLM errors, its human evaluation found these critics caught more bugs than paid human contractors. It also reports that the critics sometimes invented bugs. An ICLR 2024 paper (e2) found that LLMs struggle to correct their own reasoning without outside feedback. That finding concerns reasoning tasks, not code. Both studies measure behavior in specific settings. Neither one tests whether a model knows anything in the sense he means. Treating the claim as a practical inability to find bugs would replace it with a claim he did not make. No ordinary reading of the claim can be settled by evidence, so it cannot be graded.

Caveats

• The two studies are listed as context only. They cover specific models and tasks and do not test knowledge or understanding.
• The critic result in e1 is for models fine-tuned for code review. It may not apply to generative AI in general.
• The self-correction result in e2 is about reasoning, not code.
• Some readers may take 'know what a mistake looks like' as a practical claim about finding bugs. This assessment does not.

Evidence (2 items)

Claims like this aren’t scored, so there is no evidence to show.