Humanity’s “Oh shit!” AI moment?
We recorded 10 claims from this post. 3 were checked against evidence. 7 can’t be checked. 3 more are held. Read the post at its source.
Claims from this post (10)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“I wouldn’t take seriously any of its anthropomorphic language about LLMs “knowing” or having scheming or having intentions to deceive, etc.”
Our reading: Anthropomorphic language about LLMs knowing, scheming, or intending to deceive should not be taken seriously.
No horizon
Qualifier: in the Apollo Research scheming paper
Assessment: SupportedSaid
Observation
“They can easily tell you one thing and (especially if hooked up as agents) do another — possibly quite contrary to what they have alleged they are doing.”
Our reading: LLMs can easily say one thing and do another, especially when deployed as agents.
No horizon Evidence as of
Limitation: Follow-up results concern trials where the relevant actions had already occurred; they do not measure how often those actions occur in ordinary use [e1].
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“The important thing is that paper makes it very clear that nobody should ever take LLMs at their word.”
Our reading: Nobody should ever take LLMs at their word.
No horizon
Assessment: ContradictedSaid
Observation
“Worse, they do not actually calculate the consequences of their actions at all.”
Our reading: LLMs do not calculate the consequences of their actions at all.
No horizon Evidence as of
Limitation: The consequences were supplied explicitly in a simplified task.
Assessment: UnresolvedSaid
Observation
“They can’t reliably sanity check themselves for hallucinations, and they can’t reliably sanity check for whether what they are doing is harmful.”
Our reading: LLMs cannot reliably check themselves for hallucinations or for whether their actions are harmful.
No horizon Evidence as of
Limitation: This is an unresolved combined claim, not a finding that self-checking is dependable.
Assessment: Can’t be checkedSaid
Causal claim
“They more power we give them (but hooking up them to agents, people, etc), the more risk we are incurring.”
Our reading: Giving LLMs more power, such as hooking them up to agents, increases risk.
No horizon Evidence as of
Limitation: The only evidence item is one company's report on misuse of its own models. It records incidents and measures nothing about how harm scales with the power granted.
Assessment: Can’t be checkedSaid
Forecast
“Even if machines don’t ever rise up a la the Terminator, a bad actor could still leverage LLMs to cause massive mayhem. Possibly soon.”
Our reading: A bad actor could leverage LLMs to cause massive mayhem, possibly soon.
No horizon
Qualifier: by a bad actor
Assessment: Can’t be checkedSaid
Causal claim
“That may leave all of us extremely vulnerable to this mess.”
Our reading: That Congress achieved little and that anti-regulation AI people like Andreessen are having a good run may leave all of us extremely vulnerable to this mess.
No horizon
Qualifier: may
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“There’s a lot of anthropomorphism here in this paper that I find hard to take, but it is in other ways a smart and carefully considered paper — and ought to be a wakeup call.”
Our reading: Apollo Research's paper on in-context scheming contains a lot of anthropomorphism he finds hard to take, but is in other ways a smart and carefully considered paper and ought to be a wakeup call.
No horizon
Qualifier: in other ways
Assessment: Can’t be checkedSaid
Causal claim
“The longer we delay in regulation, the more risk we are incurring.”
Our reading: Delaying AI regulation increases the risk society incurs.
No horizon Evidence as of
Limitation: The record has no outcome evidence at all, either for or against the claim.
Held from this post (3)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews reached different assessments. Our reading, not assessed: There will be a lot of GenAI-driven cybercrime in the coming years.
- The two reviews read the claim differently
- The two reviews reached different assessments. Our reading, not assessed: Congress achieved little on AI regulation and anti-regulation AI advocates like Andreessen are succeeding.