“A lot of lawyers are using tools like ChatGPT to prepare briefs, and many still seem shocked when it makes up cases.”
Our reading:
A lot of lawyers are using tools like ChatGPT to prepare briefs, and many of them still seem shocked when it makes up cases.
Assessment: Unresolved
What happened
Was it true for the period described?
Outcome: Unresolved
We checked, and the evidence so far doesn’t settle it.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
The evidence does not settle either clause at the claimed scale during May 1–24, 2025 [e1, e2, e3, e4]. The survey published April 15 combines briefs with memos and does not separately measure brief preparation during that period [e1]. The May 20 judgment records Lee saying she was shocked when cited cases could not be found at the May 1 hearing [e2]. Tribunal findings available July 16, 2026 establish that Lee herself used ChatGPT and misled the court about its use [e4]. Those findings qualify her account of authorship without settling whether she appeared shocked [e2, e4]. Reporting on the May 21 Alabama hearing documents ChatGPT use in court filings, apologies, and acknowledged limitations, but no expression of shock [e3]. These examples establish neither substantial brief-preparation use nor many apparently shocked users; they also do not establish that either was uncommon [e1, e2, e3, e4].
Caveats
- The survey predates the target period and combines briefs with memos [e1].
- The incident records do not measure how common brief preparation or apparent shock was among lawyers [e2, e3, e4].
- One reported reaction cannot establish 'many'; the later findings about Lee's account do not settle that reaction [e2, e4].
- Apologies and acknowledged limitations alone do not establish shock or its absence [e3].
Evidence (4 items)
Supports the claim: Ontario Superior Court of Justice; copy hosted by CourtReady,
She did not have an answer to what happened and could not deal with it in the moment.
Why it matters: The court records Lee's statement that she was shocked when the cases could not be found at the May 1 hearing. This supports one instance of apparent shock, not the claimed prevalence; e4 qualifies the reliability of her surrounding explanation.
- Publisher
- Ontario Superior Court of Justice; copy hosted by CourtReady
- Kind of source
- Regulatory filing
- About events on
- Published
Context only: Thomson Reuters,
Despite these benefits, many legal professionals still have reservations about GenAI.
Why it matters: This early-2025 survey reports 59% for brief-or-memo drafting among legal professionals using the technology, but does not separate briefs from memos. Its accuracy concerns neither measure apparent shock nor exclude surprise among a substantial subgroup.
- Publisher
- Thomson Reuters
- Kind of source
- Independent measurement
- About events on
- Published
Context only: The Guardian,
“I was aware of the limitations on use [of AI], and in these two instances I did not comply with policy,” Reeves said.
Why it matters: The hearing report documents ChatGPT-generated authorities in written court arguments, but describes apologies and acknowledged limitations rather than shock. This is reporting rather than a court transcript; the date reflects the page's last modification.
- Where
- Account of the May 21 Birmingham hearing, following 'Reeves attempted to take full responsibility.'
- Publisher
- The Guardian
- Kind of source
- News report
- About events on
- Published
Context only: Law Society Tribunal; copy hosted by CourtReady,
She then deliberately misled a court multiple times when asked about her use of AI.
Why it matters: The tribunal found that Lee herself used ChatGPT and misled the court about its use, qualifying her earlier explanation blaming staff. It also found she was initially unaware the cases were hallucinations; neither finding alone proves or disproves apparent shock.
- Publisher
- Law Society Tribunal; copy hosted by CourtReady
- Kind of source
- Regulatory filing
- About events on
- Published
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Conditions
- a lot of lawyers (their words)
- many still seem shocked (their words)
Where we had to interpret
- The claim makes two commitments: substantial numbers of lawyers use generative AI to prepare briefs, and many such lawyers display apparent shock when it invents cases. Neither phrase necessarily means a majority.
- Preparing briefs includes drafting written court arguments and obtaining authorities specifically for those arguments. General legal research, correspondence, contracts, and unspecified document creation do not establish this use.
- Apparent shock requires reported expressions or observed reactions showing surprise or disbelief about fabricated cases. Failure to check citations, ignorance of risks, apologies, and embarrassment alone do not establish it.
- The target is the ongoing situation in May 2025 through publication on May 24. This uses the latest reporting month identified by the context's repeated references to examples from this month. Earlier surveys and incidents are background.
- No country is specified. Evidence from individual jurisdictions can establish examples, but any broader inference must acknowledge its coverage.
- Both clauses must hold to support the whole claim. Incomplete evidence for one clause does not establish contradiction.
Right if: Evidence establishes substantial use of ChatGPT-like tools specifically to prepare briefs during the target period, together with sufficiently broad documented reactions to establish that many users appeared shocked by fabricated cases. Surveys must distinguish briefs from other tasks; incident evidence must establish reactions, not merely fabricated citations.
Wrong if: Reliable evidence covering the relevant lawyer population establishes that brief preparation with these tools was uncommon, or that apparent shock on discovering fabricated cases was exceptional rather than occurring among many users. Failure to find enough examples is not itself contradiction.
We wrote this test after the outcome was known.
What would settle it
Task-specific evidence establishing substantial brief preparation during May 1–24, 2025, together with broad documentation that many users appeared shocked by fabricated cases, would support the claim. Reliable population evidence showing either that such brief preparation was uncommon or that apparent shock was exceptional would contradict it.
We found 10 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, AI literacy, hallucinations, and the law: A case study, paragraph 6.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-10-28. The post may have been edited after 2025-05-24.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, AI literacy, hallucinations, and the law: A case study
A lot of lawyers are using tools like ChatGPT to prepare briefs, and many still seem shocked when it makes up cases.
- Where
- paragraph 6
- Published
- 2025-05-24
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Unresolved. Evidence up to 2026-10-28.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.
Held backBy us
The final check failed: a cited source didn’t say what the write-up said.
Held backBy us
A reviewer said our test wasn’t fair to what was actually said.