“So far, it is the realists from the Class of 2023 that seem to have been best calibrated. Bugs in LLMs have stymied Khanmigo and the AI pin.”
Our reading:
So far, the realists from the TED Class of 2023 seem to have been best calibrated, and bugs in LLMs have stymied Khanmigo and the Humane AI pin.
Assessment: Supported
What happened
Does the evidence support the cause?
Outcome: Supported
The evidence matches what they said.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
The firm product claim is supported on the ordinary reading of 'stymied' as materially hindered. The Khanmigo tests describe incorrect feedback that disrupted tutoring. The AI Pin review documents failures in answering and identifying things, with its co-founder acknowledging hallucinations among the causes. Other causes also mattered, but Marcus did not say LLM bugs caused every problem. This outcome does not grade his hedged comparison of the TED speakers.
Caveats
- Support concerns impaired product functions by April 15, 2024, not complete failure or later commercial outcomes.
- Khanmigo testing is reported through a secondary source because the original was inaccessible.
- The AI Pin evidence supports contributing causation, without isolating or quantifying the model’s share of failures.
Evidence (3 items)
Supports the claim: IBL News,
frequent calculation errors
Why it matters: The account describes Khanmigo accepting incorrect arithmetic and challenging a correct answer during tutoring exercises. These errors directly interfered with its intended guidance, although this is a secondary account of the Journal’s tests.
- Publisher
- IBL News
- Kind of source
- Secondary summary
- About events on
- Published
Supports the claim: The Verge,
model hallucinations
Why it matters: The reviewer documents failed identification and incorrect answers during hands-on use; Humane’s co-founder identifies model hallucinations among the causes. The review also identifies server and hardware problems, limiting any attribution of all failures to LLMs.
- Publisher
- The Verge
- Kind of source
- Independent measurement
- About events on
- Published
Context only: Khan Academy,
errors in math
Why it matters: Khan Academy identifies GPT-4 as the basis of its experimental tutor and acknowledges mathematical errors. It also describes an operating pilot and potential benefits, so this disclosure alone does not establish material hindrance or complete failure.
- Publisher
- Khan Academy
- Kind of source
- Company disclosure: the company describing itself, not an independent finding
- About events on
- Published
How we judged it
- Kind of claim
- Causal claim
- Deadline
- None given. We don’t add one.
Conditions
- so far (their words)
- seem to have been (their words)
Where we had to interpret
- The tentative comparison that the realists 'seem' best calibrated is excluded from scoring.
- The firm claim is that LLM errors materially hindered the intended functions of both Khanmigo and the Humane AI Pin.
- The ordinary reading of 'stymied' is materially hindered, without requiring complete failure, discontinued deployment, or commercial collapse.
- LLM errors must have contributed to the hindrance; the wording does not require them to be the sole cause.
- The retrospective period ends on April 15, 2024. The wording supplies no precise lower boundary, so the start is left unspecified.
Right if: Documented use or testing by April 15, 2024 shows that erroneous model outputs materially interfered with Khanmigo’s tutoring and the Humane AI Pin’s assistance functions, with evidence connecting those problems to the models.
Wrong if: Evidence addressing the reported failures establishes that model errors did not materially hinder one or both products, or that the relevant hindrances were entirely attributable to other causes.
We wrote this test after the outcome was known.
We found 12 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, TED, AI, and the test of time, paragraph 15.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-29. The post may have been edited after 2024-04-15.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, TED, AI, and the test of time
So far, it is the realists from the Class of 2023 that seem to have been best calibrated. Bugs in LLMs have stymied Khanmigo and the AI pin.
- Where
- paragraph 15
- Published
- 2024-04-15
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Supported. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
A reviewer said our test wasn’t fair to what was actually said.