What does Meta AI’s Diplomacy-winning Cicero Mean for AI?
We recorded 10 claims from this post. 4 were checked against evidence. 6 can’t be checked. 6 more are held. Read the post at its source.
Claims from this post (10)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“The results themselves are, without question, genuinely impressive.”
Our reading: Cicero's Diplomacy results are, without question, genuinely impressive.
No horizon
Qualifier: without question
Assessment: UnresolvedSaid
Observation
“As it turns out, the architecture of Cicero differs profoundly from most of what’s been talked about in recent years in AI.”
Our reading: Cicero's architecture differs profoundly from most AI approaches discussed in recent years.
No horizon Evidence as of
Limitation: Selected examples establish differences, not prevalence.
Assessment: Can’t be checkedSaid
Causal claim
“(The contest was a “blitz” version of the game, limited to five minutes a move, which favors computers, since the blitz version is presumably less dialogue- and negotiation-intensive than the full version of the game; in a longer version of the game, humans may still dominate.)”
Our reading: The five-minute-a-move blitz version of Diplomacy favors computers, presumably because it is less dialogue- and negotiation-intensive than the full game, and in a longer version humans may still dominate.
No horizon
Qualifier: presumably
Assessment: Can’t be checkedSaid
Observation
“Cicero is in many ways a marvel; it has achieved by far the deepest and most extensive integration of language and action in a dynamic world of any AI system built to date.”
Our reading: Cicero is in many ways a marvel and has achieved by far the deepest and most extensive integration of language and action in a dynamic world of any AI system built to date.
No horizon Evidence as of
Limitation: This does not dispute that Cicero represented a substantial technical achievement.
Assessment: SupportedSaid
Observation
“Strikingly, and in opposition to much of the Zeitgeist, Cicero relies quite heavily on hand-crafting, both in the data sets, and in the architecture; in this sense it is in many ways more reminiscent of classical “Good Old Fashioned AI” than deep learning systems that tend to be less structured, and less customized to particular problems.”
Our reading: Against much of the Zeitgeist, Cicero relies quite heavily on hand-crafting in both its data sets and its architecture, and in this sense it is in many ways more reminiscent of classical Good Old Fashioned AI than of deep learning systems, which tend to be less structured and less customized to particular problems.
No horizon Evidence as of
Limitation: Quite heavily is assessed through substantive design choices, not a measured percentage of effort or performance.
Assessment: SupportedSaid
Observation
“It could not be immediately applied to the challenge of, say, customer service or guiding the actions of a domestic robot, or well, almost anything else, really.”
Our reading: Cicero cannot be immediately applied to tasks such as customer service or domestic robotics, or almost anything besides blitz Diplomacy.
No horizon Evidence as of
Limitation: The evidence is a developer capability statement, not an independent transfer benchmark.
Assessment: Can’t be checkedSaid
Conditional forecast
“Our best guess is that if you played Diplomacy with alternative rules, the system would like to have retrained almost from scratch,”
Our reading: If Diplomacy were played with alternative rules, Cicero would probably need to be retrained almost from scratch.
No horizon
Only if: if Diplomacy were played with alternative rules
Assessment: Can’t be checkedSaid
Conditional forecast
“as things stand now, the prospects for generalization seem to us to be somewhat limited; the style of work might well be useful in other problems, but it may be that not much of the specifics of the architecture would survive if the system were applied to other problems”
Our reading: Saying he cannot predict this with any great confidence, Marcus judges that, as things stand now, Cicero's prospects for generalization seem somewhat limited: its style of work might well be useful in other problems, but it may be that not much of its specific architecture would survive if the system were applied to other problems.
No horizon
Only if: if the system were applied to other problems
Assessment: SupportedSaid
Observation
“Cicero makes extensive use of machine learning, but is hardly a poster child for simply making ever bigger models (so-called “scaling maximalism”), nor for the currently popular view of “end-to-end” machine learning of in which some single general learning algorithm applies across the board, with little internal structure and zero innate knowledge.”
Our reading: Cicero makes extensive use of machine learning but is hardly a poster child for scaling maximalism or for the popular end-to-end view of machine learning with little internal structure and zero innate knowledge.
No horizon Evidence as of
Limitation: The architecture alone does not establish how much each component contributed to performance.
Assessment: Can’t be checkedSaid
Forecast
“Cicero may change that calculus.”
Our reading: Cicero may change the field's view of machine learning as a universal solvent.
No horizon
Held from this post (6)
These records did not pass every check, so nothing about them is published. Held is not a finding.