“If the system really is being “programmed in English” it is not always doing what is being asked to do:”
Our reading, the sentence that was checked:
If ChatGPT/GPT-4 really is being "programmed in English" through its allegedly leaked system prompt, it is not always doing what it is asked to do.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, The hottest new programming language is English! Or maybe not., paragraph 12.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-the-hottest-new-programming-language-042b69-a1
We recorded 4 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-02-15. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- if the system really is being programmed in English via its alleged system prompt (qualifier, stated by the speaker)
- not always (scope, stated by the speaker)
From the speaker
- If the system really is being “programmed in English”
- not always doing what is being asked to do
- internal “system prompt”
- allegedly
Our interpretation
- Preserve the condition: the relevant English instructions must actually have governed the ChatGPT sessions being evaluated.
- 'Not always' requires at least one verified violation, not a high failure rate.
- General GPT-4 failures or another product's behavior cannot establish a violation of this particular ChatGPT prompt.
- Use January 1 through February 15, 2024 as the latest completed month and elapsed publication month for this ongoing observation.
Counts as supported if: Establish that the relevant English system instructions applied, then identify at least one contemporaneous ChatGPT response or tool call that conflicts with an applicable instruction.
Counts as contradicted if: An authenticated, complete record of the relevant sessions establishes compliance and shows that the alleged violations involved different instructions, products, or misidentified outputs. A few compliant responses alone do not contradict 'not always'.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The extraction report makes the premise plausible, but the necessary link between the alleged instructions and the evaluated sessions remains missing. General evidence that models sometimes disregard instructions would not settle this more specific conditional observation. Neither the condition nor a violation under the applicable prompt is established sufficiently by the retrieved records.
Limitations
- An absent authentication record does not show that the alleged prompt was fabricated.
- Official confirmation is not essential if reliable technical evidence can establish the same facts.
- This is a conditional historical observation, not a forecast.
Evidence (2 items)
Context only: AI Changes Everything; Patrick McGuinness,
I tried it myself
Why it matters here: McGuinness reports personally extracting the purported instructions and reproduces them. This establishes a contemporaneous extraction claim, but not that those exact instructions governed the separately reported outputs.
- Publisher
- AI Changes Everything; Patrick McGuinness
- Kind of source
- Demonstration
- Describes events of
- Public from
Context only: THE DECODER; Matthias Bastian,
not been officially confirmed.
Why it matters here: The earlier report describes repeated extraction across accounts but distinguishes this from official confirmation. It concerns an earlier prompt and cannot authenticate the February version.
- Publisher
- THE DECODER; Matthias Bastian
- Kind of source
- News report
- Describes events of
- Public from
What would settle it
A reliable record establishing the instructions active in an early-2024 ChatGPT session, paired with an output or tool call that clearly violates one of them.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, The hottest new programming language is English! Or maybe not.
If the system really is being “programmed in English” it is not always doing what is being asked to do:
- Where
- paragraph 12
- Published
- 2024-02-15
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-the-hottest-new-programming-language-042b69-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-the-hottest-new-programming-language-042b69-a1.