“Strikingly, and in opposition to much of the Zeitgeist, Cicero relies quite heavily on hand-crafting, both in the data sets, and in the architecture; in this sense it is in many ways more reminiscent of classical “Good Old Fashioned AI” than deep learning systems that tend to be less structured, and less customized to particular problems.”
Our reading, the sentence that was checked:
Against much of the Zeitgeist, Cicero relies quite heavily on hand-crafting in both its data sets and its architecture, and in this sense it is in many ways more reminiscent of classical Good Old Fashioned AI than of deep learning systems, which tend to be less structured and less customized to particular problems.
Assessment: Supported Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, What does Meta AI’s Diplomacy-winning Cicero Mean for AI?, paragraph 33.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-what-does-meta-ais-diplomacy-winning-14ae8d-a1
We recorded 10 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2022-11-25. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
Conditions
- in this sense (qualifier, stated by the speaker)
- in many ways (qualifier, stated by the speaker)
- quite heavily (qualifier, stated by the speaker)
From the speaker
- relies quite heavily on hand-crafting
- both in the data sets, and in the architecture
- in this sense
- more reminiscent of classical “Good Old Fashioned AI”
- less structured, and less customized to particular problems
Our interpretation
- The plain reading is substantial deliberate, task-specific construction of training procedures and overall architecture.
- Hand-crafting includes manually specified rules for constructing synthetic examples; it does not require every resulting example to be written by a person.
- In this sense limits the classical-AI resemblance to designed structure and customization. It does not claim that Cicero lacks deep learning.
- Zeitgeist is contextual rhetoric rather than a separately quantified claim about research prevalence.
- The target is Cicero's November 2022 public release, using the release-to-post interval.
Counts as supported if: The methods document consequential task-specific data-construction rules and separately designed components or controls, beyond a generic model trained on an unmodified corpus.
Counts as contradicted if: The methods show little consequential task-specific data engineering or architectural customization, with the claimed structure instead learned through a largely generic training procedure.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Supported
The dated evidence matches what the claim committed to, under the criteria frozen before grading.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The supplement documents substantial custom data construction and a deliberately organized system of filters and strategic-planning procedures. That supports the claim about hand-crafted structure and task customization. Its classical-AI analogy is supportable in the limited sense Marcus explicitly states. Some examples were generated automatically by learned models, so the evidence does not support reading hand-crafting as manual authorship of all data or as an absence of deep learning.
Limitations
- Quite heavily is assessed through substantive design choices, not a measured percentage of effort or performance.
- The classical-AI comparison applies to customization and structure, not to a formal classification of the whole system.
- The evidence does not quantify the prevailing Zeitgeist.
Evidence (2 items)
Supports the claim: Science / AAAS; supplementary PDF hosted by Gwern,
perturbation functions
Why it matters here: The supplement specifies rules for corrupting game entities, targeted examples for counting and negation errors, and a separately configured classifier ensemble. Together with its explicit filtering and strategic-planning procedures, these document substantial task-specific engineering in both data and architecture.
- Where
- Section D.3.1, Creating counterfactuals, Entity corruptions; architectural details in D.3 and E
- Publisher
- Science / AAAS; supplementary PDF hosted by Gwern
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Context only: Science / AAAS; supplementary PDF hosted by Gwern,
this model
Why it matters here: Some custom training examples were generated by a learned model. This qualifies the wording: researchers designed the procedures and targets, but did not manually author every example.
- Publisher
- Science / AAAS; supplementary PDF hosted by Gwern
- Kind of source
- Peer-reviewed paper
- Describes events of
- Public from
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, What does Meta AI’s Diplomacy-winning Cicero Mean for AI?
Strikingly, and in opposition to much of the Zeitgeist, Cicero relies quite heavily on hand-crafting, both in the data sets, and in the architecture; in this sense it is in many ways more reminiscent of classical “Good Old Fashioned AI” than deep learning systems that tend to be less structured, and less customized to particular problems.
- Where
- paragraph 33
- Published
- 2022-11-25
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-what-does-meta-ais-diplomacy-winning-14ae8d-a1: Current. Outcome: Supported. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-what-does-meta-ais-diplomacy-winning-14ae8d-a1.