Unresolved claims by Gary Marcus, page 12 of 12

Unresolved claims by Gary Marcus (1,145)

We checked, and the evidence so far doesn’t settle it. Predictions whose deadline hasn’t come yet are under Not yet due.

Claims 1,101 to 1,145 of 1,145, newest first. Page 12 of 12.

1. Assessment: UnresolvedSaid 2022-06-22

   “Similarly, consider pronouns like “himself”: DALL-E does not do well with sentences like “The man is painting a picture of himself”.”

2. Assessment: UnresolvedSaid 2022-06-22

   “Large language models don’t produce such cognitive models, at least not in a way that anybody has been able to make reliable use of.”

3. Assessment: UnresolvedSaid 2022-06-22

   “Compositionality has long been a central concept in linguistics and philosophy (even Aristotle acknowledged it), yet so-called foundation models (large language models, GPT-3, BERT, deep learning approaches) sidestep it.”

4. Assessment: UnresolvedSaid 2022-06-22

   “(a) the tendency of large language models to lose coherence over time, drifting into “empty” language with no clear connection to reality;”

5. Assessment: UnresolvedSaid 2022-06-22

   “(b) the difficulty of large language models in distinguishing truth from falsehoods;”

6. Assessment: UnresolvedSaid 2022-06-22

   “But the interesting thing to note here is that we humans can choose to interpret language compositionally, or not—this degree of cognitive flexibility is really quite impressive, but lacking in current large language models.”

7. Assessment: UnresolvedSaid 2022-06-22

   “Language is about a connection between words (or sentence) and the world; the sequences of words that large language models utter lack connection to the external world.”

8. Assessment: UnresolvedSaid 2022-06-22

   “In such cases, DALL-E Mini typically generates images of bowls of cucumbers, and only cucumbers, leaving out the tomatoes altogether (where a human might reasonably expect there to be at least two, given the plural).”

9. Assessment: UnresolvedSaid 2022-06-22

   “The “states” that such systems visit are pretty, but lack reference, and lack ties to dynamically updated cognitive models.”

10. Assessment: UnresolvedSaid 2022-06-22

    “Unless the field can find a way to tightly integrate large language models with cognitive models and reference, we expect that the field will eventually hit a dead end, particularly in the understanding long discourses, and series of events unfolding over time, and that the problem of misinformation will remain unsolved.”

11. Assessment: UnresolvedSaid 2022-06-22

    “(c) the struggle in these models to avoid perpetuating bias and toxic speech.”

12. Assessment: UnresolvedSaid 2022-06-15

    “Does that mean that society should give up on building driverless cars? Absolutely not. In the long run, they will save lives, and also give many people, such the blind, elderly, and disabled, have far more autonomy over their own lives.”

13. Assessment: UnresolvedSaid 2022-06-15

    “As far as I can tell, nobody has a real solution to the outlier problem of coping with unusual circumstances; without it, full self driving will remain a pipe dream.”

14. Assessment: UnresolvedSaid 2022-06-12

    “We in the AI community have our differences, but pretty much all of find the notion that LaMDA might be sentient completely ridiculous.”

15. Assessment: UnresolvedSaid 2022-06-12

    “The public knows the test of course, but the AI community wishes it would go away; we all know that beating that test isn’t meaningful.”

16. Assessment: UnresolvedSaid 2022-06-11

    “👉an explicit type/token distinction will be crucial”

17. Assessment: UnresolvedSaid 2022-06-11

    “👉explicit cognitive models will be crucial”

18. Assessment: UnresolvedSaid 2022-06-11

    “👉operations over variables (including storing, retrieving and comparing values) will be crucial”

19. Assessment: UnresolvedSaid 2022-06-07

    “Large language models are (unless hybridized with symbolic systems, a la Jurassic X) are, on their own, inherently weak at understanding the physical and psychological world.”

20. Assessment: UnresolvedSaid 2022-06-07

    “Interestingly, as Ernie has just discovered, across the full set of our Technology Review questions the difference from GPT-3’s earlier version to the current version (InstructGPT) is far more modest than one might have expected: fewer outright wacky answers, some old answers now correct, but some backsliding, too.”

21. Assessment: UnresolvedSaid 2022-06-07

    “The core claims that Ernie and I made in Technology Review still all, so far as I can tell, hold true, nearly two years and many even larger models later. Yes, some of the specific examples we pointed out have been remedied, but the general problems have not.”

22. Assessment: UnresolvedSaid 2022-06-07

    “But they still don’t convey genuine comprehension, and so they are still very easy for Ernie and me (or anyone else who cares to try) to break.”

23. Assessment: UnresolvedSaid 2022-06-07

    “I am willing to bet SlateStarCodex now (terms to be negotiated) that if OpenAI gives us unrestricted access to GPT-4, whenever that is released, and assuming that is basically the same architecture but with more data, that within a day of playing around with it, Ernie and I will still be able find lots of examples of failures in physical reasoning, temporal reasoning, causal reasoning, and so forth.”

24. Assessment: UnresolvedSaid 2022-06-07

    “Now it is true that GPT-3 is genuinely better than GPT-2, and maybe (but maybe not, see footnote 1) true that InstructGPT is genuinely better than GPT-3. I do think that for any given example, the probability of a correct answer has gone up.”

25. Assessment: UnresolvedSaid 2022-06-07

    “The improvements, such as they are, come, primarily because the newer models have larger and larger sets of data about how human beings use word sequences, and bigger word sequences are certainly helpful for pattern matching machines.”

26. Assessment: UnresolvedSaid 2022-05-31

    “The fourth thing to realize is that we, and by this I mean humanity as a whole, still don’t really have an adequate methodology for building complex cognitive systems.”

27. Assessment: UnresolvedSaid 2022-05-31

    “I don’t know what you have planned for Optimus, but I can promise you that the AGI that you would need for a general-purpose domestic robot (where every home is different, and each poses its own safety risks) is way beyond what you would need for a car that drives on roads that are more or less engineered the same way from one town to the next.”

28. Assessment: UnresolvedSaid 2022-05-31

    “We still don’t have anything like stable or trustworthy solutions for common sense, reasoning, language, or analogy. Truth is, I have been using this same pie chart for about 5 years, and the situation has scarcely changed.”

29. Assessment: UnresolvedSaid 2022-05-31

    “Deep learning is pretty decent at, for example, recognizing objects, but not nearly as good at planning, or reading, or language comprehension.”

30. Assessment: UnresolvedSaid 2022-05-31

    “The thing is, it’s easy to convince yourself that AI problems are much easier than they are actually are, because of the long tail problem.”

31. Assessment: UnresolvedSaid 2022-05-28

    “As Ernie Davis, Scott Aaronson, and I showed, a week or so after the release, it’s easily fooled, especially with complex sentences and words like “similar” that would require a deep understanding of language to be interpreted correctly”

32. Assessment: UnresolvedSaid 2022-05-28

    “Since we know that Imagen can draw images of horses riding astronauts (if the instructions are specific enough) we know that the failure to draw a horse riding an astronaut given the prompt “a horse riding an astronaut” can only have one explanation: the system doesn’t understand what the prompt “a horse riding an astronaut” means.”

33. Assessment: UnresolvedSaid 2022-05-28

    “And, no, it wasn’t about common sense at all.”

34. Assessment: UnresolvedSaid 2022-05-28

    “The problem was not on the image generation side, it was in the connection between language and images.”

35. Assessment: UnresolvedSaid 2022-05-28

    “(In reality, Imagen has no idea what weight, mass, or density are.)”

36. Assessment: UnresolvedSaid 2022-05-28

    “What one really needs, and no one yet knows how to build, is a system that can derive semantics of wholes from their parts as a function of their syntax.”

37. Assessment: UnresolvedSaid 2022-05-28

    “Furthermore, there is no step in the process to try to derive such structures; so it is just wishful thinking to expect them to be fully sensitive to semantics as it relates to syntactic structure.”

38. Assessment: UnresolvedSaid 2022-05-21

    “There is no intention to create misinformation—but also no capacity to avoid it, because fundamentally, GPT-3 is a model of how words relate to one another, not a model of how language might relate to the perceived world.”

39. Assessment: UnresolvedSaid 2022-05-14

    “Virtually none of the critical problems for AI that cognitive scientists such as Steve Pinker, Judea Pearl, the late Jerry Fodor, and myself have been pointing to for decades is actually solved yet.”

40. Assessment: UnresolvedSaid 2022-05-14

    “The bottom line is this, something that AI once cherished but has now forgotten: If we are to build AGI, we are going to need to learn something from humans, how they reason and understand the physical world, and how they represent and acquire language and complex concepts.”

41. Assessment: UnresolvedSaid 2022-05-14

    “Systems like DALL-E 2, GPT-3, Flamingo, and Gato appear to be mind-blowing, but nobody who has looked at them carefully would confuse them for human intelligence.”

42. Assessment: UnresolvedSaid 2022-05-14

    “Virtually all of the world’s software, from web browsers to spreadsheets to word processors, still depends on knowledge engineering, and Sutton sells that short.”

43. Assessment: UnresolvedSaid 2022-05-14

    “So far, in most domains (not all, to be sure) Big Data has triumphed (for now) over careful knowledge engineering.”

44. Assessment: UnresolvedSaid 2022-05-14

    “As I have written, I am fine with thinking of Gato as an “Alt Intelligence”— an interesting exploration in alternative ways to build intelligence—but we need to take it in context: it doesn’t work like the brain, it doesn’t learn like a child, it doesn’t understand language, it doesn’t align with human values, and it can’t be trusted with mission-critical tasks.”

45. Assessment: UnresolvedSaid 2022-05-14

    “Right now, the predominant strand of work within Alt Intelligence is the idea of scaling.”