Did GoogleAI Just Snooker One of Silicon Valley’s Sharpest Minds?
We recorded 12 claims from this post. 7 were checked against evidence. 5 can’t be checked. 6 more are held. Read the post at its source.
Claims from this post (12)
In the order they appear in the post. Each quotation is the speaker’s words in this post. A label such as “Supported” or “Can’t be checked” is the claim’s current assessment; its page has the evidence and the reasons.
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“In 2017, a team of Google researchers invented the Transformer, arguably the biggest advance in AI in the last 10 years.”
Our reading: The Transformer, invented by a team of Google researchers in 2017, is arguably the biggest advance in AI in the last 10 years.
No horizon
Qualifier: arguably
Assessment: ContradictedSaid
Observation
“we were right to be skeptical; so far, four and half years later, Duplex hasn’t gotten a whole lot further.)”
Our reading: Four and a half years after its announcement, Google Duplex has not progressed much past its limited demo stage.
No horizon Evidence as of
Limitation: The degree of progress is qualitative.
Assessment: SupportedSaid
Observation
“The dirty secret in current AI is that for all the bloviating about how current systems have mastered human language, they are still really weak on compositionality, which most linguists (since Frege, over a century ago) would agree is at the very core of how language works.”
Our reading: Current AI systems are still very weak at compositionality, which is central to how language works.
No horizon Evidence as of
Limitation: The result depends on reading current AI as prominent contemporary neural systems rather than every AI architecture.
Assessment: Can’t be checkedSaid
Observation
“For me, compositionality has always been the touchstone for AI; when I say AI has hit a wall, it’s compositionality, more than anything else, that I have been talking about:”
Our reading: AI has hit a wall, and that wall is mainly compositionality.
No horizon
Assessment: SupportedSaid
Observation
“This should be a piece of cake, but for currently popular systems, it’s not.”
Our reading: Currently popular vision-language systems cannot reliably tell apart captions that use the same words in a different order.
No horizon Evidence as of
Limitation: The result applies to the evaluated models and task.
Assessment: UnresolvedSaid
Observation
“Notwithstanding the rumors about Google Imagen, nobody has yet publicly demonstrated a machine that can relate the meanings of sentences to their parts the way a five-year-old child can.”
Our reading: No one, Imagen included, has publicly shown a machine that relates sentence meanings to their parts as well as a five-year-old.
No horizon Evidence as of
Limitation: The named age threshold is not measured by the cited benchmark.
Assessment: UnresolvedSaid
Observation
“But many people have claimed over the years to solve the problem, and none of those systems have proven to be reliable; every proposed solution has been like Clever Hans, working in dim light, leveraging large databases to some degree, but falling apart upon careful inspection; they might get 60% on some task, but they never really master it.”
Our reading: Every claimed solution to compositionality so far has proven unreliable and falls apart on careful inspection.
No horizon Evidence as of
Limitation: The universal scope exceeds the coverage of the retrieved evaluations.
Assessment: UnresolvedSaid
Observation
“Google cloaks its current work the trappings of science (article preprints with citations and graphs and so forth) but when it comes to community inspection, they’ve systematically opted out.”
Our reading: Google has systematically refused to let the scientific community inspect models such as Imagen and LaMDA.
No horizon Evidence as of
Limitation: The evidence distinguishes public availability from scientific access but does not fully document the latter.
Assessment: Can’t be checkedSaid
Observation
“(Dare I make obvious guess? They probably tried and failed, and don’t want to admit it.)”
Our reading: Google probably tested Imagen on Winoground, got poor results, and doesn't want to admit it.
No horizon
Assessment: SupportedSaid
Observation
“The progress that Alexander is vaunting shows that the system did better on a handful of items than DALL-E2 (which was not even state of the art at compositionality when it appeared), but not that the newer system (Imagen) systematically groks compositionality”
Our reading: Alexander's test shows only that Imagen beat DALL-E 2 on a handful of items, not that Imagen systematically understands compositionality.
No horizon Evidence as of
Limitation: The scores reflect Alexander's judgments, not independent verification [e1].
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“Having a Silicon Valley celebrity like Alexander weigh in for Google is great PR for Google, but not so great for the public understanding of AI.”
Our reading: Having a Silicon Valley celebrity like Alexander weigh in for Google is great PR for Google but not so great for the public understanding of AI.
No horizon
Assessment: Can’t be checkedSaid
Recommendation or value judgment
“A will to believe in AI will never replace the need for careful science.”
Our reading: A will to believe in AI will never replace the need for careful science.
No horizon
Held from this post (6)
These records did not pass every check, so nothing about them is published. Held is not a finding.
- The two reviews read the claim differently
- The two reviews read the claim differently
- The criteria are ambiguous. Our reading, not assessed: If another AI winter comes, the cause will be hype exceeding reality rather than AI being impossible, and the only remedy is truth in advertising.
- The two reviews read the claim differently
- The record does not meet the current rules. Our reading, not assessed: Alexander's five-item test and Google's press do not change the fact that compositionality remains a giant unsolved problem for AI.
- The two reviews read the claim differently