“In many respects, this comparison is eerily prophetic, given the slowdown (and even regression) of improvement we’ve seen in LLM-based services like ChatGPT.”
Our reading, the sentence that was checked:
Improvement in LLM-based services like ChatGPT has slowed and even regressed.
Assessment: Unresolved Evidence as of
Where and when Ed Zitron wrote it
Where's Your Ed At, Managing Up, paragraph 29.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment wyea-managing-up-7dc921-a1
We recorded 5 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-28. The source may have been edited after 2024-04-29. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- slowdown
- even regression
- improvement we’ve seen
- LLM-based services like ChatGPT
Our interpretation
- The plain reading is that useful capabilities were improving more slowly, with some capabilities getting worse.
- I use 2023 through the post date as the comparison window for successive service versions. The speaker supplies neither a starting date nor a reporting period; this window is an evaluator choice.
- A slowdown requires comparisons of gains across successive intervals. A decline on one task establishes a local regression, but does not by itself establish a general slowdown.
Counts as supported if: Comparable longitudinal evaluations show diminishing gains across a representative set of capabilities, together with reproducible regressions on some tasks.
Counts as contradicted if: Comparable longitudinal evaluations show sustained or accelerating gains across representative capabilities and fail to substantiate the alleged regressions. Improvements on selected tasks alone do not disprove a slowdown.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
There is credible evidence of regressions on particular tasks and in preferences between particular versions. There is also evidence of improvements. Neither establishes the overall rate of progress across LLM services. The regression component has support, but the broader slowdown claim remains unsettled; improvements alone would not refute slower improvement.
Limitations
- The comparison window and scope are not specified by Zitron.
- Task scores, formatting compliance and user preferences measure different things.
- This assessment concerns the period before the post, not subsequent AI progress.
Evidence (3 items)
Supports the claim: arXiv; Lingjiao Chen, Matei Zaharia and James Zou,
more formatting mistakes
Why it matters here: This supports the regression component for particular service behaviors. The same abstract reports improvements on other tasks, so it does not establish a general decline or a slowing rate of improvement.
- Publisher
- arXiv; Lingjiao Chen, Matei Zaharia and James Zou
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Supports the claim: LMSYS Org,
user preference
Why it matters here: Users preferred the earlier GPT-4 version, supporting a local service regression. The authors also identify changing users and comparison sets as possible explanations; their table places GPT-4 Turbo above the June version.
- Publisher
- LMSYS Org
- Kind of source
- Benchmark result
- Describes events of
- Public from
Context only: LMSYS Org,
against the baseline
Why it matters here: The table scores April GPT-4 Turbo at 82.6 against 50.0 for the March 2023 baseline, showing improvement on this evaluation. The authors warn about model-judge bias; these comparisons do not measure whether progress was slowing.
- Publisher
- LMSYS Org
- Kind of source
- Benchmark result
- Describes events of
- Public from
What would settle it
A representative evaluation of successive service versions across multiple intervals, using stable tasks and scoring, that establishes whether capability gains were diminishing.
Where it was said (1)
Said once in the sources this register covers.
: Where's Your Ed At, Managing Up
In many respects, this comparison is eerily prophetic, given the slowdown (and even regression) of improvement we’ve seen in LLM-based services like ChatGPT.
- Where
- paragraph 29
- Published
- 2024-04-29
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment wyea-managing-up-7dc921-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.