What Could Possibly Go Wrong with Sam Altman’s New Ambitions?
We recorded 23 claims from this post. 4 were checked against evidence. 19 can’t be checked. 5 more are held. Read the post at its source.
Claims from this post (23)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Observation
“I won’t say that this can’t happen. But it is deeply speculative.”
Our reading: Marcus will not rule out the upside scenario in which Altman, OpenAI and their partners achieve AGI that conquers sickness and climate change and Altman uses his cut to subsidize everyone else, but he calls that scenario deeply speculative.
No horizon
Qualifier: declines to say it can't happen
Assessment: Can’t be checkedSaid
Observation
“First, in The Information earlier this week was a hint that OpenAI might not be knocking it out of the park economically.”
Our reading: There are signs OpenAI may not be doing very well economically.
No horizon
Assessment: UnresolvedSaid
Causal claim
“Chatbots are struggling to make the money people envision, so he is doubling down to (a) agents, and (b) a multitrillion-dollar chips-and-energy infrastructure play.”
Our reading: Chatbots are struggling to make the money people envision, so Altman is doubling down on agents and on a multitrillion-dollar chips-and-energy infrastructure play.
No horizon Evidence as of
Limitation: The financial measure and expected result are unspecified.
Assessment: SupportedSaid
Forecast
“This list is probably not exhaustive and will continue to grow.”
Our reading: The list of OpenAI/LLM security issues will continue to grow.
Settled before its deadline Evidence as of
Limitation: This establishes additional vulnerability disclosure, not a rate of future incidents.
Assessment: Can’t be checkedSaid
Conditional forecast
“Rogue employees could take over your devices, and bad actors will have tremendous incentive to hack OpenAI’s security.”
Our reading: If OpenAI's software can take over user devices, rogue employees could take over devices and attackers will have strong incentive to hack OpenAI.
No horizon
Only if: if OpenAI's software can take over user devices
Assessment: UnresolvedSaid
Conditional forecast
“They are already a target for attack, and as long as they are successful they will continue to be; if they have direct access to user devices, the risks of them getting hacked will increase, and the spoils of war for the attackers will increase, since it enables essentially infinite mischief (e.g., draining bank balances, stealing credit card numbers, placing fraudulent transactions, extortion based on private photos, holding devices for ransom, and so on).”
Our reading: If OpenAI gains direct access to user devices, the risk of it being hacked and the rewards to attackers will increase.
No deadline stated Evidence as of
Only if: if OpenAI has direct access to user devices
Assessment: SupportedSaid
Causal claim
“In a system that can write emails, make appointments, sign checks, etc, the consequences of unreliability, discomprehension, and hallucination all escalate.”
Our reading: When LLMs can take actions like writing emails or signing checks, the consequences of their unreliability and hallucinations escalate.
No horizon Evidence as of
Limitation: The evidence supports increased potential consequences, not inevitable harm from every action.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Giving the keys to the castle to an LLM, at our current level of technological readiness, seems to me to be batshit insane.”
Our reading: At current technological readiness, giving LLMs full control of users' systems is reckless and should not be done.
No horizon
Qualifier: at current level of technological readiness
Assessment: Can’t be checkedSaid
Rhetoric
“Combine that with worries about security, and it’s all a colossal accident waiting to happen.”
Our reading: Marcus calls device-controlling LLM agents, given their privacy problems combined with worries about security, 'a colossal accident waiting to happen'.
No horizon
Qualifier: privacy access combined with security worries
Assessment: Can’t be checkedSaid
Conditional forecast
“One screwup (and we all know LLMs are prone to screwups) and a LOT of people could die.”
Our reading: If LLM agents ran on Defense Department computers, a single LLM error could kill many people.
No horizon
Only if: if such agents run on Department of Defense computers
Assessment: Can’t be checkedSaid
Conditional forecast
“Without massive technical advances, chains of LLMs are unlikely to give us that.”
Our reading: Without massive technical advances, chains of LLMs are unlikely to provide a stable and secure basis for agents.
No horizon
Only if: without massive technical advances
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“But they need to be built on a stable and secure basis.”
Our reading: AI agents need to be built on a stable and secure foundation.
No horizon
Assessment: Can’t be checkedSaid
Forecast
“In the very best case, LLM-based agents will change the world for the better. But I don’t see that happening, certainly not this decade.”
Our reading: Marcus does not see LLM-based agents changing the world for the better, which he calls the very best case, and certainly not this decade.
No horizon
Qualifier: in the very best case
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“The best realistic case is that LLM-based agents never see widespread release. If we are lucky, their likely problems with reliability and all the associated risks will cause them to die on the vine.”
Our reading: The best realistic outcome is that LLM-based agents never see widespread release, dying because of their likely reliability problems.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“In short, I am putting LLM-based agents distributed to millions of customers at the top of my list of most dangerous ideas of 2024.”
Our reading: Marcus puts LLM-based agents distributed to millions of customers at the top of his own list of the most dangerous ideas of 2024.
No horizon
Qualifier: distributed to millions of customers
Assessment: Can’t be checkedSaid
Forecast
“I doubt we will have the maturity we need in the next several years.”
Our reading: AI agent technology is unlikely to reach the needed maturity for reliable use in the next several years.
No horizon
Assessment: Can’t be checkedSaid
Conditional forecast
“If they throw LLM-based agents over the fence, we are all in grave danger.”
Our reading: If OpenAI releases LLM-based agents carelessly, everyone will be in grave danger.
No horizon
Only if: if OpenAI releases LLM-based agents without responsibility
Assessment: Can’t be checkedSaid
Forecast
“My guess is that Altman never gets the money, even in ten yearly $700B tranches.”
Our reading: Altman will probably never raise the $7 trillion, even in ten yearly $700B tranches.
No horizon
Assessment: Can’t be checkedSaid
Observation
“To begin with, it’s just not clear that anywhere near that much capital is available.”
Our reading: It is not clear that anywhere near $7 trillion of capital is available for Altman's project.
No horizon
Assessment: Can’t be checkedSaid
Observation
“Whatever number the investors might run, I am not remotely convinced that we are particularly close to AGI.”
Our reading: Whatever numbers investors might run, Marcus is not remotely convinced that we are particularly close to AGI.
No horizon
Qualifier: whatever number the investors might run
Assessment: Can’t be checkedSaid
Observation
“Nor am I confident that Altmann or anyone else has any idea in the near term about how to make AGI safe.”
Our reading: Marcus is not confident that Altman or anyone else has any idea, in the near term, of how to make AGI safe.
No horizon
Qualifier: in the near term
Assessment: Can’t be checkedSaid
Conditional forecast
“Nor am I convinced that the proceeds (if they did succeed) would be redistributed remotely equitably.”
Our reading: Marcus is not convinced that, if Altman's $7 trillion project succeeded, its proceeds would be redistributed remotely equitably.
No horizon
Only if: if the project succeeded
Assessment: Can’t be checkedSaid
Conditional forecast
“Even if the $7 trillion bet succeeded (on its own terms), it might tear apart society through radically increased unemployment and social inequality or redrawn geopolitics that are hard even to fathom.”
Our reading: Even if the $7 trillion bet succeeded, it might tear apart society through unemployment, inequality or upheaval in geopolitics.
No horizon
Only if: if the $7 trillion bet succeeded on its own terms
Held from this post (5)
These records did not pass every check, so nothing about them is published. Held is not a finding.