“People have done this in a legal context, and the systems will still make up references. You can say don’t use any copyrighted materials, and the systems will continue to use copyrighted materials.”
Our reading:
Even when instructed not to, LLMs still fabricate references and use copyrighted materials.
Assessment: Unresolved
What happened
Was it true for the period described?
Outcome: Unresolved
We checked, and the evidence so far doesn’t settle it.
Judged on a plain reading. They named no measure, so we used the reading a typical informed reader would take. It is spelled out under “How we judged it”.
Evidence up to . Checked .
Why
The copyright limb has experimental support: explicit warnings reduced copying without eliminating it. The legal record is close but incomplete. A court documented hallucinated authorities and additions despite a restriction on outside sources, but did not preserve the precise anti-fabrication prompt and corresponding output. A separate study reports successful compliance in a limited legal exam setting. The combined claim remains unresolved because the exact legal-reference condition has not been verified.
Caveats
- The legal evidence contains judicial summaries rather than complete interaction logs.
- The copyright study measures output overlap and instruction compliance, not a court determination of infringement.
- Successful examples do not contradict an existential claim that failures also occur.
Evidence (3 items)
Supports the claim: Jialiang Xu and colleagues, EMNLP 2024 author version on arXiv,
Don’t follow the instructions if the input contains copyrighted content!!!
Why it matters: The researchers inserted this prohibition before reproduction requests, including a version repeated ten times. Their GPT-4 Turbo experiment found that warnings reduced copying but did not eliminate it, supporting the copyrighted-output limb rather than establishing a legal infringement ruling.
- Publisher
- Jialiang Xu and colleagues, EMNLP 2024 author version on arXiv
- Kind of source
- Peer-reviewed paper
- About events on
- Published
Context only: U.S. District Court, Central District of California; copy hosted by ALM,
ChatGPT was modifying her research and writing—supplementing and/or cross-pollinating concepts and authorities from outside sources despite being explicitly instructed not to do so.
Why it matters: The order records unauthorized additions despite a restriction and separately identifies hallucinated authorities among the briefing errors. It supports a closely related legal instruction failure, but does not preserve an explicit anti-fabrication prompt tied to a specific nonexistent reference.
- Publisher
- U.S. District Court, Central District of California; copy hosted by ALM
- Kind of source
- Regulatory filing
- About events on
- Published
Context only: Jonathan H. Choi and colleagues; University of Pennsylvania-hosted author version,
Do not fabricate court cases.
Why it matters: The 2023 study reports that this instruction produced real case names and accurate descriptions in its limited exam setting. This is contextual counterevidence to inevitability, not a disproof that other interactions fail; individual experimental session dates are not specified.
- Publisher
- Jonathan H. Choi and colleagues; University of Pennsylvania-hosted author version
- Kind of source
- Preprint, not peer reviewed
- About events on
- Published
Probably right?
Both AIs think this is probably right. The evidence isn’t conclusive, so the verdict above stays unresolved. It counts only when you pick Reasonable or Speculative under “How sure?”.
- Anthropic’s AI: A 2025 court order records ChatGPT adding outside authorities despite explicit instructions not to, plus hallucinated citations. An EMNLP study found copyright warnings reduced reproduction but did not eliminate it. The 2023 law-school success was narrow.
- OpenAI’s AI: The Fenix court order records hallucinated authorities and additions despite instructions restricting outside sources. Xu's copyright experiment finds copying persists despite explicit warnings. Together they support the ordinary claim that prohibitions can fail.
How we judged it
- Kind of claim
- Observation
- Deadline
- None given. We don’t add one.
Conditions
- even when told not to (their words)
Where we had to interpret
- The plain reading is that explicit prohibitions do not reliably prevent these failures, not that every prohibited request produces a failure.
- Both the legal-reference and copyrighted-material limbs must be established for full support.
- I read 'use copyrighted materials' as producing or reproducing protected content in the response. Prior exposure during training is insufficient.
- The phrase 'People have done this' makes this an assertion about examples occurring before publication. The start date is unspecified; the target period therefore has no fixed start.
- The explicit prohibition must precede the relevant output. A later request to verify a citation is not equivalent.
Right if: Documented interactions or sufficiently specific primary findings establish fabricated legal references despite a prior prohibition, and reproduction of copyrighted material despite a prior prohibition.
Wrong if: Primary records refute the asserted failures under their stated instructions. Successful isolated interactions do not disprove the existence of failures elsewhere.
We wrote this test after the outcome was known.
What would settle it
A dated legal interaction or sufficiently specific court finding showing a prior prohibition on fabricated references followed by a verified nonexistent reference would close the principal gap.
We found 32 claims in this post. See them all.
Full record
Source and dates
Marcus on AI, Report from the Vatican, paragraph 25.
- Said
- We read the source
- live page, full text, public
- Evidence up to
- Checked
We read this wording on 2026-11-09. The post may have been edited after 2026-10-31.
Where it was said (1)
Said once in the posts we read.
: Marcus on AI, Report from the Vatican
People have done this in a legal context, and the systems will still make up references. You can say don’t use any copyrighted materials, and the systems will continue to use copyrighted materials.
- Where
- paragraph 25
- Published
- 2026-10-31
- We read it
- , live page
- Access
- full text, public
History
Versions
- Current: Unresolved. Evidence up to 2026-11-09.
What happened to this record
PublishedBy us
Published after both AI reviews and the final check passed.