“It isn’t becoming “better” in a way that’s measurable using anything other than benchmarks invented specifically for generative AI — an industry-wide coddling of a mediocre technology that only makes money through massive subsidies, FOMO and executive ignorance.”
Our reading, the sentence that was checked:
Generative AI is not measurably improving except on AI-specific benchmarks and only makes money through subsidies, FOMO and executive ignorance.
Assessment: Unresolved Evidence as of
Where and when Ed Zitron wrote it
Where's Your Ed At, AI's Brokenomics, paragraph 87.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment wyea-brokenomics-c83df3-a1
We recorded 41 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2026-06-15. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- other than on AI-specific benchmarks (qualifier, stated by the speaker)
From the speaker
- isn’t becoming “better”
- measurable using anything other than benchmarks invented specifically for generative AI
- only makes money through massive subsidies, FOMO and executive ignorance
Our interpretation
- Becoming better means improvement over time in practical outcomes, such as completion time or output quality, beyond scores on AI-specific benchmarks.
- Evidence that AI users outperform nonusers at one point in time does not by itself establish improvement across model generations.
- Only makes money is read as a claim that commercial demand lacks an independently demonstrated economic basis outside the named mechanisms, not as a precise net-income assertion.
- For the ongoing state without a named period, I use the latest completed calendar quarter before publication, January–March 2026. Earlier measurements serve as comparison context.
- The exclusive explanation of revenue has a causal component, but the main assessable proposition is the asserted absence of measurable practical improvement.
Counts as supported if: Credible comparisons over time find no practical improvement outside excluded benchmarks, and evidence establishes that the listed mechanisms exhaust the economic explanation for commercial receipts.
Counts as contradicted if: Reliable comparisons show practical improvement over time outside AI-specific benchmarks, or documented commercial economics establish an independent basis for revenue beyond the named mechanisms.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The reviewed research points toward practical gains, but does not securely settle the improvement-over-time claim for the chosen period. METR's survey measures perceptions, while its experimental follow-up has serious selection problems. Neither establishes the exclusive explanation of revenue. These gaps prevent support, but uncertainty about the measurements is not evidence that improvement is absent.
Limitations
- The claim combines a performance trend with a sweeping explanation of commercial demand.
- Results depend on the unstated baseline and definition of better.
- The search does not establish that no qualifying study exists.
Evidence (2 items)
Context only: METR,
median 1.4–2x change
Why it matters here: The survey reports practical gains and higher retrospective estimates for March 2026 than March 2025. These are perceptions, and METR warns that responses can overstate actual gains; they do not decisively establish the claimed trend or commercial profitability.
- Publisher
- METR
- Kind of source
- Independent measurement
- Describes events of
- Public from
Context only: METR,
Our raw results show some evidence for speedup.
Why it matters here: The later-2025 experiment points toward improvement relative to the earlier experiment, but METR identifies serious selection and measurement problems. It is earlier-period context, not decisive evidence of the specified 2026-quarter outcome.
- Publisher
- METR
- Kind of source
- Independent measurement
- Describes events of
- Public from
What would settle it
A credible comparison of successive systems on comparable real work in the target period, controlling for quality and selection, could settle the performance component. Documented customer returns and provider economics would be needed to assess the commercial component.
Where it was said (1)
Said once in the sources this register covers.
: Where's Your Ed At, AI's Brokenomics
It isn’t becoming “better” in a way that’s measurable using anything other than benchmarks invented specifically for generative AI — an industry-wide coddling of a mediocre technology that only makes money through massive subsidies, FOMO and executive ignorance.
- Where
- paragraph 87
- Published
- 2026-06-15
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment wyea-brokenomics-c83df3-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.