AI has (sort of) passed the Turing Test; here’s why that hardly matters
We recorded 9 claims from this post. 4 were checked against evidence. 5 can’t be checked. 9 more are held. Read the post at its source.
Claims from this post (9)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: UnresolvedSaid
Observation
“I do genuinely believe today’s Turing Test takers are better AI systems than an earlier generation. I hardly think that means anything is “over”.”
Our reading: Today's Turing Test-taking AI systems are better than the earlier generation, but that does not mean anything is 'over'.
No horizon Evidence as of
Limitation: This is an evidence gap, not evidence that contemporary systems lack broader improvements.
Assessment: Can’t be checkedSaid
Observation
“Because, as I (and many others) have said for years, the Turing Test is a test of human gullibility, not a test of intelligence.”
Our reading: The Turing Test is a test of human gullibility, not a test of intelligence.
No horizon
Qualifier: as he and many others have said for years
Assessment: Can’t be checkedSaid
Conditional forecast
“In a faceoff, though, it might actually beat ChatGPT.”
Our reading: In a faceoff, Eugene Goostman might actually beat ChatGPT.
No horizon
Only if: in a faceoff
Assessment: SupportedSaid
Causal claim
“To do well the system had to be coached to adopt the persona of “a young person who is introverted, knowledgeable about internet culture, and uses slang”; it’s still in no small part about the tricks.”
Our reading: To do well in the new study, the system had to be coached to adopt the persona of a young, introverted, slang-using person knowledgeable about internet culture, so its success is still in no small part about tricks.
No horizon Evidence as of
Limitation: The comparison does not establish that this particular persona was necessary among all possible prompts [e1].
Assessment: Can’t be checkedSaid
Observation
“None of these systems, in my judgment, are anywhere near as intelligent as naive audiences might think.”
Our reading: None of the Turing Test chatbots, from ELIZA to ChatGPT 4.5, are anywhere near as intelligent as naive audiences might think.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Few people in the AI community take the Turing Test seriously, and few people should.”
Our reading: Few people in the AI community take the Turing Test seriously, and few should.
No horizon
Assessment: UnresolvedSaid
Observation
“Unfortunately, nobody has ever fully implemented the comprehension challenge test that I proposed.”
Our reading: Nobody has fully implemented Marcus's proposed comprehension challenge test.
No horizon Evidence as of
Limitation: The boundary between a related benchmark and full implementation is not formally defined.
Assessment: UnresolvedSaid
Conditional forecast
“But I am willing to wager that if someone put in that kind of effort and carefully built out the test, no current system would come close to human performance.”
Our reading: If a careful comprehension challenge test were built, no current AI system would come close to human performance on it.
No deadline stated Evidence as of
Only if: if someone carefully built out the comprehension challenge test
Assessment: Can’t be checkedSaid
Observation
“Fooling people for a few minutes is not the measure of intelligence, and never was.”
Our reading: Fooling people for a few minutes is not the measure of intelligence, and never was.
No horizon
Held from this post (9)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: AI has made great progress in human mimicry since 2014, but current systems still lack much in reasoning, planning, and common sense.
- The criteria are ambiguous. Our reading, not assessed: In the new study, ChatGPT without a persona prompt was beaten by ELIZA in the Turing Test.
- The criteria are ambiguous. Our reading, not assessed: Systems like Gemini can do the comprehension challenge a little but often hallucinate heavily, and he does not think they can yet take in a full Simpsons episode, let alone follow four seasons of Succession and explain in real time who is lying to whom and to what end.
- The two reviews reached different assessments. Our reading, not assessed: Those who could afford to build the comprehension challenge prefer to spend research money on scaling.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: He thinks it is fair to say that, slightly over a decade later, the 2014 Turing Test 'winner' Eugene Goostman contributed nothing of lasting value to AI.