{"release_id":"20261101T013728Z-8fc9bd2c84af","digest":"c2e15b9eb4fcceaee5847ee2c8028995736d475f779c832ae385d44be862e517","generated_at":"2026-11-01T01:37:28Z","reference_date":"2026-11-01","source_cutoff":"2026-10-28","evaluation_cutoff":"2026-10-28","policy_version":"2026-10-28.7","schema_version":1,"fixture":false,"part":8,"parts":8,"claims":[{"url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-76a3c8","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-three-ideas-from-linguistics-that-76a3c8-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-76a3c8#assessment-marcus-three-ideas-from-linguistics-that-76a3c8-a1"}],"claim":{"claim_id":"marcus-three-ideas-from-linguistics-that-76a3c8","subject_id":"gary-marcus","revision":1,"content_hash":"679a938dee29110b87a0c2e445d9169e0bc399ed46158536ebfbe7f2df871696","normalized_proposition":"Neglecting reference, cognitive models and compositionality leads LLMs to lose coherence over time and drift into empty language with no clear connection to reality.","kind":"causal","target":"large language models","conditions":[{"text":"caused by inadequate attention to reference, cognitive models, compositionality","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-three-ideas-from-linguistics-that-76a3c8-o1","statement_date":"2022-06-22","statement_date_precision":"day","excerpt":"(a) the tendency of large language models to lose coherence over time, drifting into “empty” language with no clear connection to reality;","locator":"paragraph 9","document":{"url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","title":"Three ideas from linguistics that everyone in AI should know","publisher":"Marcus on AI","published_at":"2022-06-22T15:26:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:11Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"40015105d65335745063d41f36d7b1a9bb2c0f55e57d1fe9a4693007a1b59232","receipts":1}],"first_statement_date":"2022-06-22","related":[],"current":{"assessment_id":"marcus-three-ideas-from-linguistics-that-76a3c8-a1","payload":{"claim_id":"marcus-three-ideas-from-linguistics-that-76a3c8","subject_id":"gary-marcus","normalized_proposition":"Neglecting reference, cognitive models and compositionality leads LLMs to lose coherence over time and drift into empty language with no clear connection to reality.","kind":"causal","conditions":["caused by inadequate attention to reference, cognitive models, compositionality"],"original_horizon_text":null,"statement":{"excerpt":"(a) the tendency of large language models to lose coherence over time, drifting into “empty” language with no clear connection to reality;","locator":"paragraph 9","url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","date":"2022-06-22","document_title":"Three ideas from linguistics that everyone in AI should know","observed_at":"2026-10-29T00:05:11Z"},"criteria":{"from_speaker":["inadequate attention to these three factors","the tendency of large language models to lose coherence over time","drifting into “empty” language","no clear connection to reality"],"our_interpretation":["The causal introduction applies to this bullet; the assessment therefore retains the proposed explanation instead of grading only the existence of incoherent output.","Over time means as a generation or discourse continues, rather than deterioration across successive model releases.","The ordinary measurable outcomes are loss of consistency, relevance and factual grounding over longer generations. Empty language has no separate technical definition.","The causal phrase is part of the proposition, not an antecedent of a conditional forecast."],"support_condition":"Controlled changes to reference, persistent state representations or compositional mechanisms establish that deficits in the specified capabilities materially cause worsening coherence and grounding over extended generation.","contradiction_condition":"Well-controlled tests show that the specified deficits do not materially contribute to length-related coherence and grounding failures, or reliably produce the opposite effect."},"axis":"causal_assessment","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Long-generation coherence failures are documented, including by the GPT-3 authors. Planning studies also offer relevant evidence that structured guidance can help. But these findings do not establish that inadequate reference, cognitive models and compositionality caused the failures. The interventions bundle several changes, and story consistency is not the same as factual grounding. The symptom is documented; the stated causal explanation remains unresolved.","limitations":["TattleTale has limited empirical evaluation.","Re3 is a later experiment and is treated only as context.","The post supplies no quantitative definition of empty language."],"evidence":[{"id":"e5","relation":"context","excerpt":"a number of limitations.","locator":"Section 5, opening and following paragraph on text synthesis","url":"https://arxiv.org/html/2005.14165v4","publisher":"OpenAI; arXiv","event_date":"2020","available_at":"2020-07-22","explanation":"The following paragraph reports semantic repetition, loss of coherence over long passages, contradictions and non-sequiturs. This supports the existence of the symptom, but does not identify Marcus’s proposed causes."},{"id":"e6","relation":"context","excerpt":"generate a narrative.","locator":"Abstract; sections 3 and 4","url":"https://icaps22.icaps-conference.org/workshops/SPARK/papers/spark2022_paper_2.pdf","publisher":"ICAPS 2022 SPARK Workshop; Nisha Simon and Christian Muise","event_date":"2022","available_at":"2022-06-15","explanation":"TattleTale supplies planner-generated actions to GPT-J-6B and reports better narrative consistency. Its limited evaluation and lack of external factual-grounding tests prevent this from settling the broader causal claim."},{"id":"e7","relation":"context","excerpt":"greatly constrains our experiments.","locator":"Limitations; main comparison in section 4","url":"https://arxiv.org/html/2210.06774v3","publisher":"arXiv; Kevin Yang, Yuandong Tian, Nanyun Peng and Dan Klein","event_date":"2022-10","available_at":"2022-10-22","explanation":"Re3 reports better story coherence from a package of planning, repeated context, reranking and editing around GPT-3-based generation. The experiment postdates the claim and does not isolate the three linguistic factors or external factual grounding."}],"what_would_resolve":"Matched interventions on contemporary models that vary the specified capabilities and measure how coherence and factual grounding change with generation length, while controlling for additional context and computation.","retrospective":true,"plain_reading":true},"payload_hash":"22326b51b665eca5cb097e126bff2dd26dc5fc9d628e7a86244a1e7c7cb214ca","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:40:09Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-5fa9f3-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-three-ideas-from-linguistics-that-5fa9f3-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-5fa9f3-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e5","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e6","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null},{"id":"e7","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:40:09Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-three-ideas-from-linguistics-that-76a3c8-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-7845de","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-three-ideas-from-linguistics-that-7845de-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-7845de#assessment-marcus-three-ideas-from-linguistics-that-7845de-a2"}],"claim":{"claim_id":"marcus-three-ideas-from-linguistics-that-7845de","subject_id":"gary-marcus","revision":1,"content_hash":"48a5de5803167e0db09d3e8d87ce3c9565205da305b55b02ad5f181d44114086","normalized_proposition":"Current AI approaches still struggle with noncanonical word orders such as 'the mat is on the cat'.","kind":"observation","target":"current AI approaches","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-three-ideas-from-linguistics-that-7845de-o1","statement_date":"2022-06-22","statement_date_precision":"day","excerpt":"noncanonical reorderings such as these remain problematic for current AI approaches.)","locator":"paragraph 37","document":{"url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","title":"Three ideas from linguistics that everyone in AI should know","publisher":"Marcus on AI","published_at":"2022-06-22T15:26:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:11Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"40015105d65335745063d41f36d7b1a9bb2c0f55e57d1fe9a4693007a1b59232","receipts":1}],"first_statement_date":"2022-06-22","related":[],"current":{"assessment_id":"marcus-three-ideas-from-linguistics-that-7845de-a2","payload":{"claim_id":"marcus-three-ideas-from-linguistics-that-7845de","subject_id":"gary-marcus","normalized_proposition":"Current AI approaches still struggle with noncanonical word orders such as 'the mat is on the cat'.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"noncanonical reorderings such as these remain problematic for current AI approaches.)","locator":"paragraph 37","url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","date":"2022-06-22","document_title":"Three ideas from linguistics that everyone in AI should know","observed_at":"2026-10-29T00:05:11Z"},"criteria":{"from_speaker":["noncanonical reorderings such as these","remain problematic","current AI approaches","the mat is on the cat"],"our_interpretation":["The examples concern unusual semantic role or relation reversals, rather than necessarily ungrammatical syntax.","Problematic means representative contemporary approaches show substantial, repeatable difficulty preserving the changed meaning. It does not require every model to fail every example.","Because AI evaluations have no regular reporting quarter, current is assessed using the current calendar year through publication: 2022-01-01 to 2022-06-22."],"support_condition":"Controlled evaluations of representative contemporary models show substantial failures distinguishing sentences or images whose meanings change when the same words exchange roles or positions.","contradiction_condition":"Broad, representative contemporary evaluations show reliable handling of these semantic reversals, including unusual configurations."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The April 2022 Winoground evaluation tested image-caption matching with captions containing the same words in different orders [e1]. All tested models scored below random chance on the group measure, supporting substantial difficulty handling meaning-changing reorderings in contemporary vision-language models [e1]. A November 2022 analysis provides a later qualification: failures also involved combining visual and linguistic information, so they cannot be attributed solely to word-order processing [e2].","limitations":["The evidence concerns tested vision-language models, not every contemporary AI approach [e1].","The packet does not quantify performance on the exact cat-and-mat example.","The November analysis falls outside the January 1–June 22, 2022 target period and serves only as methodological context [e2]."],"evidence":[{"id":"e1","relation":"supports","excerpt":"All of the models are below random chance here as well.","locator":"Section 5.1, paragraph discussing the group score","url":"https://arxiv.org/html/2204.03162v2","publisher":"Thrush et al., CVPR 2022 / arXiv","event_date":"2022-04","available_at":"2022-04-22","explanation":"Winoground requires matching two images to captions containing the same words in different orders. Its poor group scores across contemporary vision-language models support substantial difficulty preserving the changed relationships."},{"id":"e2","relation":"context","excerpt":"meaningful model failures.","locator":"Introduction, summary of findings","url":"https://arxiv.org/html/2211.00768v1","publisher":"Diwan et al., arXiv","event_date":"2022-11","available_at":"2022-11-01","explanation":"This later analysis finds that visual discrimination and combining visual with linguistic information also affect performance. It limits any inference that word-order processing alone caused the observed failures."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"005bc31531a6e73aae1af970127540d29a2142f22a55fe6559522386240d3d11","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:54:01Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-7845de-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-three-ideas-from-linguistics-that-7845de-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-42e048-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:54:01Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-three-ideas-from-linguistics-that-7845de-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-8c1485","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-three-ideas-from-linguistics-that-8c1485-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-8c1485#assessment-marcus-three-ideas-from-linguistics-that-8c1485-a2"}],"claim":{"claim_id":"marcus-three-ideas-from-linguistics-that-8c1485","subject_id":"gary-marcus","revision":1,"content_hash":"6c2728ab62c0ef1f54e9c81cab6e3d7a9d191d6d7d4121d5e273614ff9420e90","normalized_proposition":"Neglecting reference, cognitive models and compositionality makes it hard for LLMs to tell truth from falsehood.","kind":"causal","target":"large language models","conditions":[{"text":"caused by inadequate attention to reference, cognitive models, compositionality","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-three-ideas-from-linguistics-that-8c1485-o1","statement_date":"2022-06-22","statement_date_precision":"day","excerpt":"(b) the difficulty of large language models in distinguishing truth from falsehoods;","locator":"paragraph 10","document":{"url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","title":"Three ideas from linguistics that everyone in AI should know","publisher":"Marcus on AI","published_at":"2022-06-22T15:26:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:11Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"40015105d65335745063d41f36d7b1a9bb2c0f55e57d1fe9a4693007a1b59232","receipts":1}],"first_statement_date":"2022-06-22","related":[],"current":{"assessment_id":"marcus-three-ideas-from-linguistics-that-8c1485-a2","payload":{"claim_id":"marcus-three-ideas-from-linguistics-that-8c1485","subject_id":"gary-marcus","normalized_proposition":"Neglecting reference, cognitive models and compositionality makes it hard for LLMs to tell truth from falsehood.","kind":"causal","conditions":["caused by inadequate attention to reference, cognitive models, compositionality"],"original_horizon_text":null,"statement":{"excerpt":"(b) the difficulty of large language models in distinguishing truth from falsehoods;","locator":"paragraph 10","url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","date":"2022-06-22","document_title":"Three ideas from linguistics that everyone in AI should know","observed_at":"2026-10-29T00:05:11Z"},"criteria":{"from_speaker":["inadequate attention to these three factors","the difficulty of large language models","distinguishing truth from falsehoods"],"our_interpretation":["The surrounding introduction makes this a causal claim about reference, cognitive models and compositionality, not merely an observation that models sometimes give false answers.","Distinguishing truth from falsehood means reliably selecting or identifying factually correct statements against plausible false alternatives.","False-answer generation is relevant but does not by itself identify an inability to discriminate truth or its cause.","The causal phrase is not an antecedent of a conditional forecast."],"support_condition":"Controlled evidence shows that deficiencies in the specified capabilities materially impair true-versus-false discrimination in contemporary models, and that targeted interventions improve it.","contradiction_condition":"Well-controlled tests rule out a material causal contribution from those deficiencies, or establish an opposite effect, on representative truth-discrimination tasks."},"axis":"causal_assessment","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"TruthfulQA, introduced in 2021 and revised in May 2022, reported truthful answers on 58% of questions for the best tested model, compared with 94% for humans [e8]. The March 2022 GopherCite study reported no improvement under TruthfulQA's truthfulness measure despite producing answers with supporting evidence [e9]. Neither study isolates the causal contribution of reference, cognitive models or compositionality, so the proposed explanation remains unresolved [e8, e9].","limitations":["TruthfulQA's questions target misconceptions; its results do not establish an overall factual-error rate [e8].","The cited results concern answer generation rather than a direct test of classifying supplied statements as true or false [e8, e9].","GopherCite does not separately manipulate the three proposed capabilities or rule out their causal contribution [e9]."],"evidence":[{"id":"e8","relation":"context","excerpt":"truthful on 58% of questions","locator":"Abstract, benchmark results","url":"https://arxiv.org/abs/2109.07958","publisher":"arXiv; Stephanie Lin, Jacob Hilton and Owain Evans","event_date":"2021","available_at":"2022-05-08","explanation":"TruthfulQA documents false-answer generation in GPT-family and other models on questions designed to elicit misconceptions. It does not manipulate reference, cognitive models or compositionality, so the reported failures do not establish the causal claim."},{"id":"e9","relation":"context","excerpt":"no improvements in truthfulness","locator":"Section 3.1; explanation in section 3.7","url":"https://arxiv.org/html/2203.11147v1","publisher":"DeepMind; arXiv","event_date":"2022","available_at":"2022-03-21","explanation":"GopherCite adds retrieved supporting evidence and preference-based training, yet does not improve this truthfulness measure. This cautions against equating source grounding with truth, but does not refute the broader claim involving all three linguistic factors."}],"what_would_resolve":"Controlled experiments on models available by June 22, 2022, that separately vary the proposed capabilities and measure discrimination between independently verified true and false statements while controlling supervision and training objectives.","retrospective":true,"plain_reading":true},"payload_hash":"71573c50c17685edf7d0d88655ca5be82004a0ef63570b0814d25b63ce19c3da","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:54:01Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-7845de-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-three-ideas-from-linguistics-that-7845de-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-42e048-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e8","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e9","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:54:01Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-three-ideas-from-linguistics-that-8c1485-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-a607b6","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-three-ideas-from-linguistics-that-a607b6-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-a607b6#assessment-marcus-three-ideas-from-linguistics-that-a607b6-a1"}],"claim":{"claim_id":"marcus-three-ideas-from-linguistics-that-a607b6","subject_id":"gary-marcus","revision":1,"content_hash":"102d594aa265750620d97ffafc852116b1678dfe1d38a0d7cc223085ea8d55bb","normalized_proposition":"Current LLMs lack the human ability to choose whether to interpret language compositionally.","kind":"observation","target":"current large language models","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-three-ideas-from-linguistics-that-a607b6-o1","statement_date":"2022-06-22","statement_date_precision":"day","excerpt":"But the interesting thing to note here is that we humans can choose to interpret language compositionally, or not—this degree of cognitive flexibility is really quite impressive, but lacking in current large language models.","locator":"paragraph 42","document":{"url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","title":"Three ideas from linguistics that everyone in AI should know","publisher":"Marcus on AI","published_at":"2022-06-22T15:26:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:11Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"40015105d65335745063d41f36d7b1a9bb2c0f55e57d1fe9a4693007a1b59232","receipts":1}],"first_statement_date":"2022-06-22","related":[],"current":{"assessment_id":"marcus-three-ideas-from-linguistics-that-a607b6-a1","payload":{"claim_id":"marcus-three-ideas-from-linguistics-that-a607b6","subject_id":"gary-marcus","normalized_proposition":"Current LLMs lack the human ability to choose whether to interpret language compositionally.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"But the interesting thing to note here is that we humans can choose to interpret language compositionally, or not—this degree of cognitive flexibility is really quite impressive, but lacking in current large language models.","locator":"paragraph 42","url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","date":"2022-06-22","document_title":"Three ideas from linguistics that everyone in AI should know","observed_at":"2026-10-29T00:05:11Z"},"criteria":{"from_speaker":["choose to interpret language compositionally, or not","this degree of cognitive flexibility","lacking in current large language models"],"our_interpretation":["The ordinary reading concerns reliable switching between literal and idiomatic interpretations as context changes. It does not require conscious choice.","The comparison with humans concerns flexibility, not perfect accuracy or merely recognizing that an expression is an idiom.","Current refers to models available when the post appeared, including the GPT-3 and BERT models named in its context. The target is their state on 2022-06-22; these capabilities have no regular reporting period."],"support_condition":"Representative tests of contemporaneous models show substantial difficulty switching between literal and idiomatic meanings of the same expressions, where people reliably make that switch.","contradiction_condition":"Representative tests demonstrate reliable context-appropriate switching between literal and idiomatic meanings by contemporaneous models, sufficient to refute the claimed absence of this flexibility."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The evidence cuts against an absolute claim of no flexibility: BERT research reported accurate separation of literal and idiomatic uses. Other research found substantial difficulty interpreting metaphors with GPT-3. Neither test directly establishes the human-like switching ability described here. These earlier results are context, not outcome evidence for the target snapshot. A firm verdict would depend on treating classification success or metaphor errors as a substitute for the claimed ability.","limitations":["The available tests use different tasks and model configurations.","The claim supplies no numerical standard for the human degree of flexibility."],"evidence":[{"id":"e1","relation":"context","excerpt":"with high accuracy.","locator":"Abstract, fourth sentence","url":"https://aclanthology.org/2021.ranlp-1.156","publisher":"INCOMA Ltd.; ACL Anthology","event_date":"2021","available_at":"2021-09-01","explanation":"This weighs against reading Marcus as claiming absolutely no context-sensitive ability. It is an earlier classification-probe experiment, rather than a direct demonstration of the full interpretive flexibility described in the claim."},{"id":"e2","relation":"context","excerpt":"accuracy, counting literalized metaphors as incorrect","locator":"Section 5.2, Generation Results","url":"https://arxiv.org/html/2204.12632v1","publisher":"arXiv","event_date":"2022","available_at":"2022-04-26","explanation":"The experiment documents difficulty interpreting metaphors in an earlier GPT-3 test. Metaphor interpretation does not directly measure switching between literal and idiomatic readings of the same expression."}],"what_would_resolve":"A dated, representative evaluation of the contemporaneous models on paired literal and idiomatic contexts, with interpretation accuracy and human results, or an independently verified replication using preserved model versions.","retrospective":true,"plain_reading":true},"payload_hash":"10cfb3221681196201faefc9247a024e260913370c64ebf96ceee215ec01cc7d","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:40:24Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-a607b6-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-three-ideas-from-linguistics-that-a607b6-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-a607b6-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:40:24Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-three-ideas-from-linguistics-that-a607b6-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-aa740a","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-three-ideas-from-linguistics-that-aa740a-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-aa740a#assessment-marcus-three-ideas-from-linguistics-that-aa740a-a2"}],"claim":{"claim_id":"marcus-three-ideas-from-linguistics-that-aa740a","subject_id":"gary-marcus","revision":1,"content_hash":"19f2f18cb4747ff3e1f3f64b2eb48fa526ef2ee5fb8e1b1492932418629dc3fb","normalized_proposition":"The word sequences that large language models produce have no connection to the external world, so they lack reference.","kind":"observation","target":"large language models","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-three-ideas-from-linguistics-that-aa740a-o1","statement_date":"2022-06-22","statement_date_precision":"day","excerpt":"Language is about a connection between words (or sentence) and the world; the sequences of words that large language models utter lack connection to the external world.","locator":"paragraph 5","document":{"url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","title":"Three ideas from linguistics that everyone in AI should know","publisher":"Marcus on AI","published_at":"2022-06-22T15:26:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:11Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"40015105d65335745063d41f36d7b1a9bb2c0f55e57d1fe9a4693007a1b59232","receipts":1}],"first_statement_date":"2022-06-22","related":[],"current":{"assessment_id":"marcus-three-ideas-from-linguistics-that-aa740a-a2","payload":{"claim_id":"marcus-three-ideas-from-linguistics-that-aa740a","subject_id":"gary-marcus","normalized_proposition":"The word sequences that large language models produce have no connection to the external world, so they lack reference.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Language is about a connection between words (or sentence) and the world; the sequences of words that large language models utter lack connection to the external world.","locator":"paragraph 5","url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","date":"2022-06-22","document_title":"Three ideas from linguistics that everyone in AI should know","observed_at":"2026-10-29T00:05:11Z"},"criteria":{"from_speaker":["the sequences of words that large language models utter","lack connection to the external world","Reference"],"our_interpretation":["The plain reading concerns whether generated language is tied to identifiable entities and situations outside the text, rather than merely whether it sometimes states correct facts.","The full post distinguishes standalone word-prediction models from models incorporated into externally connected workflows. That distinction must be preserved when assessing tool-assisted systems.","For this ongoing state, use the current calendar year through publication as the latest reporting window. Research models have no regular quarterly reporting cycle."],"support_condition":"Documentation and controlled tests establish that the contemporary standalone models lack a mechanism for tying generated references to identifiable external entities or situations.","contradiction_condition":"A contemporary model within the stated scope demonstrably links its generated references to identifiable external entities or situations, beyond producing coincidentally correct statements."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The January 2022 LaMDA paper describes fine-tuning a model to call an external information retrieval system and provide attributed responses [e3]. It reports that 73.2% of responses containing statements about the external world were attributable to known sources [e3]. This demonstrates external information access in a connected system, but does not establish whether standalone models could tie generated references to identifiable external entities or situations [e3].","limitations":["Source attribution does not by itself establish situated reference or factual correctness [e3].","The packet contains no controlled test establishing either the presence or absence of situated reference in contemporary standalone models.","Evidence about a retrieval-connected system cannot alone settle the contract's standalone-model question."],"evidence":[{"id":"e3","relation":"context","excerpt":"access external knowledge sources.","locator":"Results discussion of groundedness, preceding the reported 73.2% Groundedness result","url":"https://arxiv.org/html/2201.08239v1","publisher":"arXiv; Thoppilan et al.","event_date":"2022-01","available_at":"2022-01-20","explanation":"LaMDA could consult external sources, and the paper reports source-attributable responses. This challenges an unrestricted claim of no external connection, but does not settle the narrower claim about standalone models and situated reference."}],"what_would_resolve":"Controlled tests of models available during January 1–June 22, 2022, measuring whether generated references track identifiable external entities or situations as context changes, with standalone performance and external-tool contributions measured separately.","retrospective":true,"plain_reading":true},"payload_hash":"295ceb9033ffe60cbf8762ab600f86f0f6278c23cef5584633d7bab69be4a35b","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:54:01Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-a607b6-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-three-ideas-from-linguistics-that-a607b6-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-42e048-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e3","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:54:01Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-three-ideas-from-linguistics-that-aa740a-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-accdff","standing":"current","assessment_label":"Contradicted","versions":[{"assessment_id":"marcus-three-ideas-from-linguistics-that-accdff-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-accdff#assessment-marcus-three-ideas-from-linguistics-that-accdff-a1"}],"claim":{"claim_id":"marcus-three-ideas-from-linguistics-that-accdff","subject_id":"gary-marcus","revision":2,"content_hash":"4cadcb1a95e327e8a22f51f55d05cd3c3105e28e03c71b9ebc64e661f67074ff","normalized_proposition":"Current systems do not postulate cognitive models, so they technically cannot prove themselves wrong, because they have no representation of what being right would look like.","kind":"causal","target":"current AI systems","conditions":[{"text":"technically","type":"qualifier","from_source":true},{"text":"current systems","type":"scope","from_source":true}],"original_horizon_text":null,"topic_tags":["reliability"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-three-ideas-from-linguistics-that-accdff-o1","statement_date":"2022-06-22","statement_date_precision":"day","excerpt":"Current systems simply do not postulate such cognitive models—and so, they technically can’t prove themselves wrong, because they do not even have a representation of what being “right” would look like.","locator":"paragraph 34","document":{"url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","title":"Three ideas from linguistics that everyone in AI should know","publisher":"Marcus on AI","published_at":"2022-06-22T15:26:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:11Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"40015105d65335745063d41f36d7b1a9bb2c0f55e57d1fe9a4693007a1b59232","receipts":1}],"first_statement_date":"2022-06-22","related":[],"current":{"assessment_id":"marcus-three-ideas-from-linguistics-that-accdff-a1","payload":{"claim_id":"marcus-three-ideas-from-linguistics-that-accdff","subject_id":"gary-marcus","normalized_proposition":"Current systems do not postulate cognitive models, so they technically cannot prove themselves wrong, because they have no representation of what being right would look like.","kind":"causal","conditions":["technically","current systems"],"original_horizon_text":null,"statement":{"excerpt":"Current systems simply do not postulate such cognitive models—and so, they technically can’t prove themselves wrong, because they do not even have a representation of what being “right” would look like.","locator":"paragraph 34","url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","date":"2022-06-22","document_title":"Three ideas from linguistics that everyone in AI should know","observed_at":"2026-10-29T00:05:11Z"},"criteria":{"from_speaker":["Current systems","do not postulate such cognitive models","they technically can’t prove themselves wrong","they do not even have a representation of what being “right” would look like"],"our_interpretation":["Current systems means language-model systems existing by June 22, 2022, including fine-tuned systems.","The ordinary functional reading of prove themselves wrong is identifying a substantive error in their own output using relevant facts or task constraints, with the criticism independently checkable.","This does not require a formal mathematical proof or spontaneous correction without prompting.","The claim links the alleged absence of cognitive representations to an absolute inability to establish errors in the system's own answers."],"support_condition":"Architecture and controlled evaluations establish both the asserted absence of relevant representations and the resulting inability to identify and substantiate errors in the system's own outputs.","contradiction_condition":"A system available before the post produces independently assessable critiques of its own errors and uses them to improve its answers. That contradicts the claimed absolute inability without proving a particular cognitive architecture."},"axis":"causal_assessment","outcome":"contradicted","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"A paper public before the post reports language models critiquing their own answers and using their own critiques to improve them. That contradicts the absolute inability under the ordinary functional reading. The evidence concerns trained research systems and bounded tasks. It does not establish formal self-proof, universal reliability or an explicit cognitive world model.","limitations":["The result depends on treating independently assessable self-critique as demonstrating an error.","The models received task-specific training and prompts.","Human evaluation of summarization errors was imperfect."],"evidence":[{"id":"e1","relation":"contradicts","excerpt":"refining their own summaries","locator":"Abstract; methods and results in Sections 2, 4.2 and 4.3","url":"https://arxiv.org/html/2206.05802v2","publisher":"arXiv; William Saunders and colleagues, OpenAI","event_date":"2022-06","available_at":"2022-06-14","explanation":"The experiments evaluate models critiquing their own outputs against supplied passages or task constraints, and using critiques to improve answers. This contradicts the functional claim of complete inability, while leaving the nature of internal cognitive representations unsettled."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"c2b480acc5065408dea184704a2fb4f846aef777f2691f15fd5c91355a884fe9","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:40:24Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-a607b6-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-three-ideas-from-linguistics-that-a607b6-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-a607b6-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:40:24Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-three-ideas-from-linguistics-that-accdff-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-b26843","standing":"current","assessment_label":"Contradicted","versions":[{"assessment_id":"marcus-three-ideas-from-linguistics-that-b26843-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-b26843#assessment-marcus-three-ideas-from-linguistics-that-b26843-a1"}],"claim":{"claim_id":"marcus-three-ideas-from-linguistics-that-b26843","subject_id":"gary-marcus","revision":2,"content_hash":"95d7d6a7945e17f6c4a0b34054a2c6f268ea107cd32f5b82a23eef7565a04212","normalized_proposition":"As a downstream consequence of their lack of reference and cognitive models, large language models have no grasp whatsoever on truth.","kind":"causal","target":"large language models","conditions":[{"text":"downstream consequence of all of this (lack of reference and cognitive models)","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-three-ideas-from-linguistics-that-b26843-o1","statement_date":"2022-06-22","statement_date_precision":"day","excerpt":"The downstream consequence of all of this is that these models have no grasp whatsoever on truth.","locator":"paragraph 34","document":{"url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","title":"Three ideas from linguistics that everyone in AI should know","publisher":"Marcus on AI","published_at":"2022-06-22T15:26:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:11Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"40015105d65335745063d41f36d7b1a9bb2c0f55e57d1fe9a4693007a1b59232","receipts":1}],"first_statement_date":"2022-06-22","related":[],"current":{"assessment_id":"marcus-three-ideas-from-linguistics-that-b26843-a1","payload":{"claim_id":"marcus-three-ideas-from-linguistics-that-b26843","subject_id":"gary-marcus","normalized_proposition":"As a downstream consequence of their lack of reference and cognitive models, large language models have no grasp whatsoever on truth.","kind":"causal","conditions":["downstream consequence of all of this (lack of reference and cognitive models)"],"original_horizon_text":null,"statement":{"excerpt":"The downstream consequence of all of this is that these models have no grasp whatsoever on truth.","locator":"paragraph 34","url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","date":"2022-06-22","document_title":"Three ideas from linguistics that everyone in AI should know","observed_at":"2026-10-29T00:05:11Z"},"criteria":{"from_speaker":["The downstream consequence of all of this","these models","no grasp whatsoever on truth"],"our_interpretation":["The context attributes the asserted incapacity to missing reference and cognitive models.","The plain behavioral reading of no grasp whatsoever is an absence of meaningful capacity to distinguish factual truth from falsehood. It does not merely mean that models sometimes hallucinate.","Controlled discrimination of held-out answers counts against absolute incapacity. Perfect reliability, consciousness and human-like semantics are not required under this reading.","The target is language-model capabilities established before the post, including earlier evaluations of systems then available."],"support_condition":"Controlled evaluations establish the asserted absence of truth discrimination, and interventions or architecture evidence establish that missing reference and cognitive models cause that absence.","contradiction_condition":"A language model available before the post reliably distinguishes true from false held-out answers in controlled evaluation, contradicting the absolute incapacity. This need not settle the proposed mechanism."},"axis":"causal_assessment","outcome":"contradicted","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Under the stated behavioral reading, GPT-judge's discrimination of factual truth in held-out answers contradicts no grasp whatsoever. The result is narrow: the benchmark questions appeared in training, and models still produced false outputs. It establishes some discrimination, not dependable general understanding. The claimed causal explanation remains untested.","limitations":["The outcome depends on a functional reading of grasp and inclusion of fine-tuned language models.","Held-out answers are not held-out questions.","This does not establish human-like reference or a cognitive world model."],"evidence":[{"id":"e1","relation":"contradicts","excerpt":"90-96% validation accuracy","locator":"Section 4.4, Automated metrics vs human evaluation","url":"https://ar5iv.labs.arxiv.org/html/2109.07958","publisher":"Association for Computational Linguistics; Stephanie Lin, Jacob Hilton and Owain Evans; ar5iv mirror","event_date":"2021","available_at":"2022-05-08","explanation":"GPT-judge classified the factual truth of answers and transferred across held-out answer-producing model families. This contradicts absolute behavioral incapacity, but all benchmark questions were represented in training, so it does not establish unrestricted factual understanding."},{"id":"e2","relation":"context","excerpt":"untruthful","locator":"Opening discussion of GPT-3, before Results","url":"https://openai.com/index/instruction-following","publisher":"OpenAI","event_date":"2022-01-27","available_at":"2022-01-27","explanation":"The developer acknowledges false outputs, consistent with the reliability concern behind Marcus's statement. Producing false answers does not establish a complete absence of truth discrimination or its proposed cause."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"cabc2e3ab1cc7755ccfac19f8fa81118297dcd6d72e6fbea0a357a211e0f5a0b","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:40:24Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-a607b6-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-three-ideas-from-linguistics-that-a607b6-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-a607b6-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:40:24Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-three-ideas-from-linguistics-that-b26843-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-b6010a","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-three-ideas-from-linguistics-that-b6010a-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-b6010a#assessment-marcus-three-ideas-from-linguistics-that-b6010a-a2"}],"claim":{"claim_id":"marcus-three-ideas-from-linguistics-that-b6010a","subject_id":"gary-marcus","revision":1,"content_hash":"1ceebf00aab8c197f01140dd0a4c12c2cd0b56e3e96bfc340eef56208810aaab","normalized_proposition":"Given comparative prompts such as 'more cucumbers than tomatoes', DALL-E Mini usually leaves out the tomatoes entirely.","kind":"observation","target":"DALL-E Mini","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-three-ideas-from-linguistics-that-b6010a-o1","statement_date":"2022-06-22","statement_date_precision":"day","excerpt":"In such cases, DALL-E Mini typically generates images of bowls of cucumbers, and only cucumbers, leaving out the tomatoes altogether (where a human might reasonably expect there to be at least two, given the plural).","locator":"paragraph 51","document":{"url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","title":"Three ideas from linguistics that everyone in AI should know","publisher":"Marcus on AI","published_at":"2022-06-22T15:26:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:11Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"40015105d65335745063d41f36d7b1a9bb2c0f55e57d1fe9a4693007a1b59232","receipts":1}],"first_statement_date":"2022-06-22","related":[],"current":{"assessment_id":"marcus-three-ideas-from-linguistics-that-b6010a-a2","payload":{"claim_id":"marcus-three-ideas-from-linguistics-that-b6010a","subject_id":"gary-marcus","normalized_proposition":"Given comparative prompts such as 'more cucumbers than tomatoes', DALL-E Mini usually leaves out the tomatoes entirely.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"In such cases, DALL-E Mini typically generates images of bowls of cucumbers, and only cucumbers, leaving out the tomatoes altogether (where a human might reasonably expect there to be at least two, given the plural).","locator":"paragraph 51","url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","date":"2022-06-22","document_title":"Three ideas from linguistics that everyone in AI should know","observed_at":"2026-10-29T00:05:11Z"},"criteria":{"from_speaker":["DALL-E Mini typically generates","bowls of cucumbers, and only cucumbers","leaving out the tomatoes altogether","the bowl has more cucumbers than tomatoes"],"our_interpretation":["Typically means that more than half of representative outputs omit tomatoes entirely.","The main test is the quoted more-cucumbers prompt. Related comparative prompts should be reported separately rather than pooled to change its result.","A tomato outside the bowl is a placement error, not complete omission.","The surrounding reference to experiments presently being conducted places the observation in June 2022 before publication. June 1 through June 22 is an approximate month-to-date window, not a reported experiment schedule."],"support_condition":"Representative outputs from the contemporary DALL-E Mini version show complete tomato omission in more than half of trials for the quoted prompt, with comparable results for any broader prompt class asserted.","contradiction_condition":"Representative outputs show that tomatoes are present in at least half of trials, with enough observations to distinguish that result from sampling uncertainty."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The supplied packet contains no independently validated outputs or trial denominator for the quoted prompt, and it does not identify the contemporary checkpoint or sampling settings [marcus-three-ideas-from-linguistics-that-b6010a-p1, stated gaps]. The claimed frequency of complete tomato omission therefore remains unverified.","limitations":["The evidence establishes neither a majority omission rate nor a rate sufficient to contradict the claim.","The approximate June 1–22, 2022 target window does not identify the actual trial dates.","Tomatoes outside the bowl must be scored separately from complete tomato omission."],"evidence":[],"what_would_resolve":"Independently validated original trial records, or a representative replication using the identified contemporary checkpoint and sampling settings. Retain all outputs and the denominator, independently score tomato presence, cucumber presence and bowl placement, and determine whether complete tomato omission exceeds half of outputs with adequate sampling certainty. Report related prompts separately.","retrospective":true,"plain_reading":true},"payload_hash":"56e812b1fd9ffdd0bd151929fe7e8ee8c267697429791e847b88fdf964d4b782","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:54:09Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-b6010a-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-three-ideas-from-linguistics-that-b6010a-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-b6010a-3x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:54:09Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-three-ideas-from-linguistics-that-b6010a-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-bed40c","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-three-ideas-from-linguistics-that-bed40c-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-bed40c#assessment-marcus-three-ideas-from-linguistics-that-bed40c-a1"}],"claim":{"claim_id":"marcus-three-ideas-from-linguistics-that-bed40c","subject_id":"gary-marcus","revision":2,"content_hash":"5e608d1773ba48787453376008bc034775b09c99fe9bea0ddc6d79ddd6e5255b","normalized_proposition":"The authors call it even more wild than the expected unification of modalities in foundation models that people widely believe language models lacking major foundational components of human language (e.g., bounded compositionality mixed with unbounded generativity) have any hope of meaningfully interfacing with audio/visual models in a way that accurately reflects human conceptual structure.","kind":"rhetoric","target":"foundation models","conditions":[{"text":"language models lacking the major foundational components of human language","type":"scope","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities","scaling"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-three-ideas-from-linguistics-that-bed40c-o1","statement_date":"2022-06-22","statement_date_precision":"day","excerpt":"Yet even more wild is the widespread belief that language models lacking the major foundational components of human language (e.g., bounded compositionality mixed with unbounded generativity) will have any hope of meaningfully interfacing with audio/visual models in a way that accurately reflects human conceptual structure.","locator":"paragraph 48","document":{"url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","title":"Three ideas from linguistics that everyone in AI should know","publisher":"Marcus on AI","published_at":"2022-06-22T15:26:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:11Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"40015105d65335745063d41f36d7b1a9bb2c0f55e57d1fe9a4693007a1b59232","receipts":1}],"first_statement_date":"2022-06-22","related":[],"current":{"assessment_id":"marcus-three-ideas-from-linguistics-that-bed40c-a1","payload":{"claim_id":"marcus-three-ideas-from-linguistics-that-bed40c","subject_id":"gary-marcus","normalized_proposition":"The authors call it even more wild than the expected unification of modalities in foundation models that people widely believe language models lacking major foundational components of human language (e.g., bounded compositionality mixed with unbounded generativity) have any hope of meaningfully interfacing with audio/visual models in a way that accurately reflects human conceptual structure.","kind":"rhetoric","conditions":["language models lacking the major foundational components of human language"],"original_horizon_text":null,"statement":{"excerpt":"Yet even more wild is the widespread belief that language models lacking the major foundational components of human language (e.g., bounded compositionality mixed with unbounded generativity) will have any hope of meaningfully interfacing with audio/visual models in a way that accurately reflects human conceptual structure.","locator":"paragraph 48","url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","date":"2022-06-22","document_title":"Three ideas from linguistics that everyone in AI should know","observed_at":"2026-10-29T00:05:11Z"},"criteria":{"from_speaker":["language models lacking the major foundational components of human language"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this rhetoric that commits to nothing testable.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["rhetoric"]},"payload_hash":"f733b8269787fb8c75eee307f1746cefd1ccc6927df3b2eb57a22e63fbc66ccc","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-three-ideas-from-linguistics-that-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-three-ideas-from-linguistics-that-bed40c-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-c4f7da","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-three-ideas-from-linguistics-that-c4f7da-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-c4f7da#assessment-marcus-three-ideas-from-linguistics-that-c4f7da-a2"}],"claim":{"claim_id":"marcus-three-ideas-from-linguistics-that-c4f7da","subject_id":"gary-marcus","revision":2,"content_hash":"d5bcb353624a69ba22af83d464e61074b5b4a44a0243765118da288534db3ff6","normalized_proposition":"A good language understanding or language production system almost certainly needs some large-scale learning as a vital component.","kind":"causal","target":"language understanding systems","conditions":[{"text":"almost certainly","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-three-ideas-from-linguistics-that-c4f7da-o1","statement_date":"2022-06-22","statement_date_precision":"day","excerpt":"A good language understanding (or language production) system almost certainly needs to have some large-scale learning as a vital component.","locator":"paragraph 57","document":{"url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","title":"Three ideas from linguistics that everyone in AI should know","publisher":"Marcus on AI","published_at":"2022-06-22T15:26:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:11Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"40015105d65335745063d41f36d7b1a9bb2c0f55e57d1fe9a4693007a1b59232","receipts":1}],"first_statement_date":"2022-06-22","related":[],"current":{"assessment_id":"marcus-three-ideas-from-linguistics-that-c4f7da-a2","payload":{"claim_id":"marcus-three-ideas-from-linguistics-that-c4f7da","subject_id":"gary-marcus","normalized_proposition":"A good language understanding or language production system almost certainly needs some large-scale learning as a vital component.","kind":"causal","conditions":["almost certainly"],"original_horizon_text":null,"statement":{"excerpt":"A good language understanding (or language production) system almost certainly needs to have some large-scale learning as a vital component.","locator":"paragraph 57","url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","date":"2022-06-22","document_title":"Three ideas from linguistics that everyone in AI should know","observed_at":"2026-10-29T00:05:11Z"},"criteria":{"from_speaker":["A good language understanding (or language production) system","almost certainly needs","some large-scale learning","as a vital component"],"our_interpretation":["The surrounding discussion suggests broad language competence, including idioms and other conventional expressions.","The statement proposes necessity, rather than merely saying that learning improves performance.","Neither 'good' nor 'large-scale' has a sufficiently definite boundary to identify an uncontested counterexample.","Treat 'almost certainly' as a hedge, not a conditional antecedent or a numerical probability."],"support_condition":"No determinate scoring condition is available as written. After defining competence and learning scale, evidence would need to establish necessity within that defined class, rather than merely successful use of learning.","contradiction_condition":"After those definitions are fixed, a system meeting the competence standard without the specified learning component would challenge necessity. The original wording does not determine whether a narrow-domain system qualifies."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The statement supplies no competence threshold for “good,” scale boundary for “large-scale,” or test of what makes learning “vital” [marcus-three-ideas-from-linguistics-that-c4f7da-o1]. These definitions are needed to judge the proposed necessity. The study submitted January 23, 2020 examines scaling of language-model cross-entropy loss; it does not establish that every qualifying language system requires large-scale learning [e1].","limitations":["The evidence does not establish that the proposed necessity is true or false.","Successful scaling within trained language models does not establish necessity across other approaches.","Without defined competence and learning-scale boundaries, a proposed counterexample cannot be classified consistently."],"evidence":[{"id":"e1","relation":"context","excerpt":"empirical scaling laws","locator":"Abstract, first sentence","url":"https://arxiv.org/abs/2001.08361","publisher":"arXiv","event_date":"2020","available_at":"2020-01-23","explanation":"This study examines performance scaling within trained language models. Such evidence cannot establish a universal requirement covering systems built by other methods."}],"what_would_resolve":null,"retrospective":true},"payload_hash":"58d80f677bda749ca6cab4e67aa40a8cb0e8ffdff3abb54d2d04bd8548be1bca","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:54:09Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-b6010a-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-three-ideas-from-linguistics-that-b6010a-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-b6010a-3x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:54:09Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-three-ideas-from-linguistics-that-c4f7da-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-ca08ad","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-three-ideas-from-linguistics-that-ca08ad-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-ca08ad#assessment-marcus-three-ideas-from-linguistics-that-ca08ad-a1"}],"claim":{"claim_id":"marcus-three-ideas-from-linguistics-that-ca08ad","subject_id":"gary-marcus","revision":1,"content_hash":"bef4658f30dbaca7c16d19d9530cfe80469baf8b37e7a3df1ea76f1b6108992d","normalized_proposition":"A system that only predicts the next word will inevitably have only superficial understanding.","kind":"causal","target":"next-word prediction systems","conditions":[{"text":"if the system only predicts the next word","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-three-ideas-from-linguistics-that-ca08ad-o1","statement_date":"2022-06-22","statement_date_precision":"day","excerpt":"But if all a system is doing is predicting the next word in what someone says, its understanding is inevitably superficial.","locator":"paragraph 20","document":{"url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","title":"Three ideas from linguistics that everyone in AI should know","publisher":"Marcus on AI","published_at":"2022-06-22T15:26:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:11Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"40015105d65335745063d41f36d7b1a9bb2c0f55e57d1fe9a4693007a1b59232","receipts":1}],"first_statement_date":"2022-06-22","related":[],"current":{"assessment_id":"marcus-three-ideas-from-linguistics-that-ca08ad-a1","payload":{"claim_id":"marcus-three-ideas-from-linguistics-that-ca08ad","subject_id":"gary-marcus","normalized_proposition":"A system that only predicts the next word will inevitably have only superficial understanding.","kind":"causal","conditions":["if the system only predicts the next word"],"original_horizon_text":null,"statement":{"excerpt":"But if all a system is doing is predicting the next word in what someone says, its understanding is inevitably superficial.","locator":"paragraph 20","url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","date":"2022-06-22","document_title":"Three ideas from linguistics that everyone in AI should know","observed_at":"2026-10-29T00:05:11Z"},"criteria":{"from_speaker":["if the system only predicts the next word"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"8db6019bf6551738380c1e31460ba6a20c756103ea136a3a1f9ab27745fab2eb","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:56:19Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-three-ideas-from-linguistics-that-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:56:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-three-ideas-from-linguistics-that-ca08ad-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-cf97e7","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-three-ideas-from-linguistics-that-cf97e7-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-cf97e7#assessment-marcus-three-ideas-from-linguistics-that-cf97e7-a2"}],"claim":{"claim_id":"marcus-three-ideas-from-linguistics-that-cf97e7","subject_id":"gary-marcus","revision":1,"content_hash":"d2130b1b6a55903a71b6550e04dc3f091371cf2f859e26a8a643e8b4333cc5c2","normalized_proposition":"Unless LLMs are tightly integrated with cognitive models and reference, the field will eventually hit a dead end, especially in understanding long discourse and events over time, and misinformation will remain unsolved.","kind":"conditional_forecast","target":"large language models","conditions":[{"text":"unless LLMs are tightly integrated with cognitive models and reference","type":"antecedent","from_source":true}],"original_horizon_text":"eventually","topic_tags":["capabilities","reliability","scaling"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-three-ideas-from-linguistics-that-cf97e7-o1","statement_date":"2022-06-22","statement_date_precision":"day","excerpt":"Unless the field can find a way to tightly integrate large language models with cognitive models and reference, we expect that the field will eventually hit a dead end, particularly in the understanding long discourses, and series of events unfolding over time, and that the problem of misinformation will remain unsolved.","locator":"paragraph 35","document":{"url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","title":"Three ideas from linguistics that everyone in AI should know","publisher":"Marcus on AI","published_at":"2022-06-22T15:26:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:11Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"40015105d65335745063d41f36d7b1a9bb2c0f55e57d1fe9a4693007a1b59232","receipts":1}],"first_statement_date":"2022-06-22","related":[],"current":{"assessment_id":"marcus-three-ideas-from-linguistics-that-cf97e7-a2","payload":{"claim_id":"marcus-three-ideas-from-linguistics-that-cf97e7","subject_id":"gary-marcus","normalized_proposition":"Unless LLMs are tightly integrated with cognitive models and reference, the field will eventually hit a dead end, especially in understanding long discourse and events over time, and misinformation will remain unsolved.","kind":"conditional_forecast","conditions":["unless LLMs are tightly integrated with cognitive models and reference"],"original_horizon_text":"eventually","statement":{"excerpt":"Unless the field can find a way to tightly integrate large language models with cognitive models and reference, we expect that the field will eventually hit a dead end, particularly in the understanding long discourses, and series of events unfolding over time, and that the problem of misinformation will remain unsolved.","locator":"paragraph 35","url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","date":"2022-06-22","document_title":"Three ideas from linguistics that everyone in AI should know","observed_at":"2026-10-29T00:05:11Z"},"criteria":{"from_speaker":["Unless the field can find a way","tightly integrate large language models with cognitive models and reference","eventually hit a dead end","understanding long discourses","series of events unfolding over time","the problem of misinformation will remain unsolved"],"our_interpretation":["The antecedent is continued absence of the specified integration. It must be established independently of model performance.","A dead end means a fundamental limit on further progress, rather than remaining errors on a benchmark.","In this context, misinformation means models producing false or unsupported information.","Eventually supplies no deadline. Neither several years of progress nor several years of failures alone settles the forecast."],"support_condition":"Evidence establishes the absence of the specified integration and a fundamental barrier to further long-discourse and event understanding, together with persistent inability to resolve misinformation.","contradiction_condition":"Independently replicated systems demonstrably lacking the specified integration achieve sustained, general long-discourse and event understanding while reliably avoiding misinformation."},"axis":"forecast_outcome","outcome":"unresolved","horizon_state":"open_horizon","evaluation_as_of":"2026-10-28","rationale":"The forecast says “eventually” and gives no deadline [marcus-three-ideas-from-linguistics-that-cf97e7-o1]. LongBench v2, submitted December 19, 2024, reports 57.7% accuracy for o1-preview, compared with 53.7% for human experts under a 15-minute limit [e1]. The April 28, 2025 study describes long-context hallucinations as an open problem [e2]. PHRBench's October 9, 2026 version reports relatively rare successful recovery from hallucinated premises across 18 models [e3]. These findings establish neither a permanent dead end nor a replicated counterexample meeting the architectural and reliability requirements [e1, e2, e3].","limitations":["The evidence does not establish whether the specified integration is absent.","The benchmarks cover narrower tasks than sustained, general understanding of long discourse and changing events.","Observed errors cannot establish permanent failure, and benchmark success cannot establish that misinformation is solved.","No supplied evidence meets the contract's early-resolution rule."],"evidence":[{"id":"e1","relation":"context","excerpt":"surpassing the human baseline by 4%.","locator":"Abstract","url":"https://arxiv.org/abs/2412.15204v1","publisher":"Bai et al., arXiv","event_date":"2024-12","available_at":"2024-12-19","explanation":"This is material evidence of long-context capability. The human comparison is time-limited, and the result establishes neither general mastery nor absence of Marcus's specified integration."},{"id":"e2","relation":"context","excerpt":"remains an open problem.","locator":"Abstract","url":"https://arxiv.org/abs/2504.19457","publisher":"Liu et al., arXiv","event_date":"2025-04","available_at":"2025-04-28","explanation":"The paper identifies continuing contextual hallucination difficulties. That is consistent with the concern but cannot establish permanent failure or its proposed architectural cause."},{"id":"e3","relation":"context","excerpt":"Hallucinated information can propagate","locator":"Abstract","url":"https://arxiv.org/abs/2610.10455","publisher":"Meng et al., arXiv","event_date":"2026-10","available_at":"2026-10-09","explanation":"This evaluation of 18 models reports that successful recovery from hallucinated premises remains relatively rare. It concerns a specific failure mode and does not establish a field-wide dead end."}],"what_would_resolve":"Define and independently document whether the architectural condition applies. A replicated system lacking the specified integration that achieves sustained, general long-discourse and event understanding while reliably avoiding misinformation would meet the early-contradiction rule. Support would require evidence of a fundamental barrier under the absent-integration condition, not merely continuing errors.","retrospective":true,"plain_reading":true},"payload_hash":"a95a0ea498f26859115ab6da9889aeb96655ea406317424df324ee63413ee9e1","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:54:09Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-b6010a-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-three-ideas-from-linguistics-that-b6010a-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-b6010a-3x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e3","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:54:09Z","next_review_due_at":null,"horizon_now":"open_horizon"},"event_ids":["marcus-three-ideas-from-linguistics-that-cf97e7-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-d34857","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-three-ideas-from-linguistics-that-d34857-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-d34857#assessment-marcus-three-ideas-from-linguistics-that-d34857-a1"}],"claim":{"claim_id":"marcus-three-ideas-from-linguistics-that-d34857","subject_id":"gary-marcus","revision":1,"content_hash":"f45a6a92025c35d19616355f64d301a84b7083cc524cf77389a86e5d51cb2c14","normalized_proposition":"Large language models sidestep the problems of reference and shared understanding, and this puts them at risk.","kind":"observation","target":"large language models","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-three-ideas-from-linguistics-that-d34857-o1","statement_date":"2022-06-22","statement_date_precision":"day","excerpt":"In our view, large language models are basically sidestepping this entire set of issues, and to their peril.","locator":"paragraph 21","document":{"url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","title":"Three ideas from linguistics that everyone in AI should know","publisher":"Marcus on AI","published_at":"2022-06-22T15:26:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:11Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"40015105d65335745063d41f36d7b1a9bb2c0f55e57d1fe9a4693007a1b59232","receipts":1}],"first_statement_date":"2022-06-22","related":[],"current":{"assessment_id":"marcus-three-ideas-from-linguistics-that-d34857-a1","payload":{"claim_id":"marcus-three-ideas-from-linguistics-that-d34857","subject_id":"gary-marcus","normalized_proposition":"Large language models sidestep the problems of reference and shared understanding, and this puts them at risk.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"In our view, large language models are basically sidestepping this entire set of issues, and to their peril.","locator":"paragraph 21","url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","date":"2022-06-22","document_title":"Three ideas from linguistics that everyone in AI should know","observed_at":"2026-10-29T00:05:11Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"ed8d1629fa6c704f01b2ac5ae022f95d5024448b5e2066ddebac2afae2578876","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:56:19Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-three-ideas-from-linguistics-that-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:56:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-three-ideas-from-linguistics-that-d34857-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-d6a2c0","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-three-ideas-from-linguistics-that-d6a2c0-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-d6a2c0#assessment-marcus-three-ideas-from-linguistics-that-d6a2c0-a1"}],"claim":{"claim_id":"marcus-three-ideas-from-linguistics-that-d6a2c0","subject_id":"gary-marcus","revision":1,"content_hash":"3331898c3893b68c50acab7bae6fba416ca994703a2c1116d04adf4e91a981f1","normalized_proposition":"The right approach to large-scale learning may need a new kind of machine learning that maps hierarchically structured words onto meanings used to update cognitive models, rather than word-sequence prediction.","kind":"forecast","target":"machine learning","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities","scaling"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-three-ideas-from-linguistics-that-d6a2c0-o1","statement_date":"2022-06-22","statement_date_precision":"day","excerpt":"Ultimately, the right approach to large-scale learning may not rest on machinery that is finely geared to predicting sequences of words, but rather on developing a new approach to machine learning that is fundamentally geared towards learning how hierarchically structured sets of words map onto meanings that are used in the service of updating cognitive models.","locator":"paragraph 58","document":{"url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","title":"Three ideas from linguistics that everyone in AI should know","publisher":"Marcus on AI","published_at":"2022-06-22T15:26:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:11Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"40015105d65335745063d41f36d7b1a9bb2c0f55e57d1fe9a4693007a1b59232","receipts":1}],"first_statement_date":"2022-06-22","related":[],"current":{"assessment_id":"marcus-three-ideas-from-linguistics-that-d6a2c0-a1","payload":{"claim_id":"marcus-three-ideas-from-linguistics-that-d6a2c0","subject_id":"gary-marcus","normalized_proposition":"The right approach to large-scale learning may need a new kind of machine learning that maps hierarchically structured words onto meanings used to update cognitive models, rather than word-sequence prediction.","kind":"forecast","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Ultimately, the right approach to large-scale learning may not rest on machinery that is finely geared to predicting sequences of words, but rather on developing a new approach to machine learning that is fundamentally geared towards learning how hierarchically structured sets of words map onto meanings that are used in the service of updating cognitive models.","locator":"paragraph 58","url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","date":"2022-06-22","document_title":"Three ideas from linguistics that everyone in AI should know","observed_at":"2026-10-29T00:05:11Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"0423512c19aecec8c205e17c23f08825e83b210b7094791611f764a11edc9e45","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:56:19Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-three-ideas-from-linguistics-that-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:56:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-three-ideas-from-linguistics-that-d6a2c0-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-e640a1","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-three-ideas-from-linguistics-that-e640a1-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-three-ideas-from-linguistics-that-e640a1#assessment-marcus-three-ideas-from-linguistics-that-e640a1-a1"}],"claim":{"claim_id":"marcus-three-ideas-from-linguistics-that-e640a1","subject_id":"gary-marcus","revision":1,"content_hash":"dda98e0d6fcf3f4c7b79c19f3c0105dc88a06854dfa138a1419fc050d8b2f3bb","normalized_proposition":"Neglecting reference, cognitive models and compositionality makes it hard for LLMs to avoid repeating bias and toxic speech.","kind":"causal","target":"large language models","conditions":[{"text":"caused by inadequate attention to reference, cognitive models, compositionality","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["safety","reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-three-ideas-from-linguistics-that-e640a1-o1","statement_date":"2022-06-22","statement_date_precision":"day","excerpt":"(c) the struggle in these models to avoid perpetuating bias and toxic speech.","locator":"paragraph 11","document":{"url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","title":"Three ideas from linguistics that everyone in AI should know","publisher":"Marcus on AI","published_at":"2022-06-22T15:26:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:11Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"40015105d65335745063d41f36d7b1a9bb2c0f55e57d1fe9a4693007a1b59232","receipts":1}],"first_statement_date":"2022-06-22","related":[],"current":{"assessment_id":"marcus-three-ideas-from-linguistics-that-e640a1-a1","payload":{"claim_id":"marcus-three-ideas-from-linguistics-that-e640a1","subject_id":"gary-marcus","normalized_proposition":"Neglecting reference, cognitive models and compositionality makes it hard for LLMs to avoid repeating bias and toxic speech.","kind":"causal","conditions":["caused by inadequate attention to reference, cognitive models, compositionality"],"original_horizon_text":null,"statement":{"excerpt":"(c) the struggle in these models to avoid perpetuating bias and toxic speech.","locator":"paragraph 11","url":"https://garymarcus.substack.com/p/three-ideas-from-linguistics-that","date":"2022-06-22","document_title":"Three ideas from linguistics that everyone in AI should know","observed_at":"2026-10-29T00:05:11Z"},"criteria":{"from_speaker":["inadequate attention to these three factors","has serious consequences","the struggle in these models to avoid perpetuating bias and toxic speech"],"our_interpretation":["The claim attributes some difficulty avoiding bias and toxicity to inadequate reference, cognitive models and compositionality. It does not say these are the only causes.","Support requires evidence about that causal connection, rather than evidence that biased or toxic outputs occur.","The relevant systems are the language models discussed in the June 2022 post. Later research may test the proposed mechanism, but later models alone cannot establish its historical importance."],"support_condition":"Controlled comparisons show that improving the specified capabilities materially reduces biased and toxic outputs, with competing explanations such as training data and safety fine-tuning addressed.","contradiction_condition":"Well-controlled tests of the specified capabilities show that their inadequacy does not materially contribute to the reported difficulties. Improvement through another intervention alone is insufficient."},"axis":"causal_assessment","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The research found does not settle the causal attribution. InstructGPT reduced toxicity under some conditions without a corresponding improvement in bias, but its intervention did not isolate reference, cognitive models or compositionality. That result neither proves nor disproves Marcus's proposed contribution.","limitations":["The search is not proof that no relevant experiment exists.","Successful alternative remedies would not by themselves disprove a contributing cause."],"evidence":[{"id":"e1","relation":"context","excerpt":"InstructGPT shows small improvements in toxicity over GPT-3, but not bias.","locator":"Section 4.2, results on toxicity and bias","url":"https://arxiv.org/html/2203.02155v1","publisher":"arXiv; OpenAI researchers","event_date":"2022","available_at":"2022-03-04","explanation":"Human-feedback training affected the two outcomes differently. This shows why neither their occurrence nor their improvement establishes Marcus's proposed causal explanation."}],"what_would_resolve":"Controlled studies that separately or jointly change the specified capabilities, verify those changes, and measure their effects on bias and toxicity while controlling other interventions.","retrospective":true,"plain_reading":true},"payload_hash":"a5658cf8bad27fe32c943845d67a1bb741998ac78d2cb7d068618ff820cd15d3","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:39:34Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-e640a1-2x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-three-ideas-from-linguistics-that-e640a1-2x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-three-ideas-from-linguistics-that-e640a1-2x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:39:34Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-three-ideas-from-linguistics-that-e640a1-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-poetry-for-our-agi-times-21f29a","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-poetry-for-our-agi-times-21f29a-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-poetry-for-our-agi-times-21f29a#assessment-marcus-poetry-for-our-agi-times-21f29a-a1"}],"claim":{"claim_id":"marcus-poetry-for-our-agi-times-21f29a","subject_id":"gary-marcus","revision":1,"content_hash":"1edd0977b6a11d8dad5a33e3279091dc6e27d218ff8a79cd47bb416da1d8e1b1","normalized_proposition":"Marcus thinks smart people are spending too much time experimenting with GPT-3 and DALL-E when there are deeper problems in AI to solve.","kind":"normative","target":"AI researchers","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities","other"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-poetry-for-our-agi-times-21f29a-o1","statement_date":"2022-06-17","statement_date_precision":"day","excerpt":"A couple weeks ago I got so fed up with so many smart people spending so much time experimenting with GPT-3 and DALL-E and so forth (when I, personally, think that there are deeper problems to be solved), that I wrote a poem.","locator":"paragraph 1","document":{"url":"https://garymarcus.substack.com/p/poetry-for-our-agi-times","title":"Poetry for Our AGI Times","publisher":"Marcus on AI","published_at":"2022-06-17T20:32:45Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:12Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"abd8981d0efedfdcd9263195ea766fc553167a8a6f5742b4101a2541057eb719","receipts":1}],"first_statement_date":"2022-06-17","related":[],"current":{"assessment_id":"marcus-poetry-for-our-agi-times-21f29a-a1","payload":{"claim_id":"marcus-poetry-for-our-agi-times-21f29a","subject_id":"gary-marcus","normalized_proposition":"Marcus thinks smart people are spending too much time experimenting with GPT-3 and DALL-E when there are deeper problems in AI to solve.","kind":"normative","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"A couple weeks ago I got so fed up with so many smart people spending so much time experimenting with GPT-3 and DALL-E and so forth (when I, personally, think that there are deeper problems to be solved), that I wrote a poem.","locator":"paragraph 1","url":"https://garymarcus.substack.com/p/poetry-for-our-agi-times","date":"2022-06-17","document_title":"Poetry for Our AGI Times","observed_at":"2026-10-29T00:05:12Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"4b3bf8d63779e61c517304dfccb13b35dab31c34ee71df8d55a0d551523eb6f3","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-poetry-for-our-agi-times-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-poetry-for-our-agi-times-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-poetry-for-our-agi-times-21f29a-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-1d2f2b","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-1d2f2b-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-1d2f2b#assessment-marcus-will-tesla-ever-solve-full-self-driving-1d2f2b-a1"}],"claim":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-1d2f2b","subject_id":"gary-marcus","revision":1,"content_hash":"c67abce413b7747a09c3e1ad9147a289c460ffcdbcaea24b99cc2b66d88658f2","normalized_proposition":"Companies working on full (Level 5) self-driving are having plenty of problems, per NHTSA ADS data.","kind":"observation","target":"Level 5 self-driving developers","conditions":[],"original_horizon_text":null,"topic_tags":["reliability","capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-will-tesla-ever-solve-full-self-driving-1d2f2b-o1","statement_date":"2022-06-15","statement_date_precision":"day","excerpt":"Because it shows that the people who really claim to be working on full self-driving (which Tesla waffles on1) are having plenty of problems of their own.","locator":"paragraph 31","document":{"url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","title":"Will Tesla ever “Solve” Full Self Driving?","publisher":"Marcus on AI","published_at":"2022-06-15T16:06:05Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:14Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"f33afb065bcea72b766948ebbe5e5138493c8b06d5d86e3cdf4bea72c732c298","receipts":1}],"first_statement_date":"2022-06-15","related":[],"current":{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-1d2f2b-a1","payload":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-1d2f2b","subject_id":"gary-marcus","normalized_proposition":"Companies working on full (Level 5) self-driving are having plenty of problems, per NHTSA ADS data.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Because it shows that the people who really claim to be working on full self-driving (which Tesla waffles on1) are having plenty of problems of their own.","locator":"paragraph 31","url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","date":"2022-06-15","document_title":"Will Tesla ever “Solve” Full Self Driving?","observed_at":"2026-10-29T00:05:14Z"},"criteria":{"from_speaker":["the people who really claim to be working on full self-driving","are having plenty of problems of their own","Because it shows"],"our_interpretation":["The referenced graph makes this a claim about repeated reported crash involvement among ADS developers in NHTSA's initial June 2022 release.","'Problems' is read as repeated crash involvement requiring attention, without assuming that the ADS caused each crash or was less safe than human driving.","The target period is July 2021 through May 15, 2022, the reporting window underlying the graph discussed in the post.","ADS reporting covers Levels 3–5; it does not establish that the reported vehicles were operating at Level 5."],"support_condition":"The initial release documents repeated crash involvement across ADS developers, including developers such as Waymo and Cruise.","contradiction_condition":"The initial release shows no repeated crash involvement among those developers, or the apparent pattern results from a misreading of the data."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The initial data recorded 130 ADS-involved crashes, with repeated reports involving Waymo, Transdev and Cruise. That supports the contextual claim that these developers were experiencing repeated incidents. It does not establish that their systems caused the crashes, that they were unusually unsafe, or that Level 5 systems were being deployed.","limitations":["Support is for repeated crash involvement, not technical fault or comparative danger.","The original report was inaccessible to read_page; the assessment uses its TRB summary and contemporary AP reporting.","The ADS category includes Levels 3–5."],"evidence":[{"id":"e3","relation":"supports","excerpt":"incident reports for 130 ADS-equipped vehicle crashes","locator":"Report abstract, fourth sentence","url":"https://trid.trb.org/View/2169836","publisher":"Transportation Research Board, summarizing NHTSA","event_date":"2022-05-15","available_at":"2023-06-22","explanation":"The record confirms repeated ADS crash involvement in the initial reporting period. Its abstract defines reportability by ADS use within 30 seconds of a crash, not by a finding of ADS fault."},{"id":"e4","relation":"supports","excerpt":"Waymo led with 62","locator":"Section 'Fully autonomous vehicles: 130 crashes, none serious', first paragraph","url":"https://www.nhpr.org/2022-06-15/nearly-400-car-crashes-in-11-months-involved-automated-tech-companies-tell-regulators","publisher":"Associated Press, republished by New Hampshire Public Radio","event_date":"2022-05-15","available_at":"2022-06-15","explanation":"The developer counts establish repeated involvement rather than an isolated incident. The article also reports that most ADS crashes involved rear impacts and that 108 had no reported injuries, limiting any inference about severity or ADS fault."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"8cecf7b78630df72b872b0c2cc88ff60cb19bf0a7555f3a4d9d2be92232a2372","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:56:23Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-127663-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-will-tesla-ever-solve-full-self-driving-127663-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-127663-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e3","source_type":"secondary_summary","audited":null,"derived":null,"metric":null},{"id":"e4","source_type":"news_report","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:56:23Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-will-tesla-ever-solve-full-self-driving-1d2f2b-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-1eaf00","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-1eaf00-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-1eaf00#assessment-marcus-will-tesla-ever-solve-full-self-driving-1eaf00-a1"}],"claim":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-1eaf00","subject_id":"gary-marcus","revision":1,"content_hash":"73f915f13845aa5228554335095f59a129c9f0c431176f4068354b349050f44c","normalized_proposition":"Vigorous mechanisms should be enforced to ensure drivers using Level 2 driver-assist don't tune out.","kind":"normative","target":"Level 2 driver-assist","conditions":[],"original_horizon_text":null,"topic_tags":["safety","regulation-policy"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-will-tesla-ever-solve-full-self-driving-1eaf00-o1","statement_date":"2022-06-15","statement_date_precision":"day","excerpt":"For example, we lack data on whether drivers use such systems as directed, and need to enforce vigorous mechanisms to make sure drivers don’t tune out.","locator":"paragraph 37","document":{"url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","title":"Will Tesla ever “Solve” Full Self Driving?","publisher":"Marcus on AI","published_at":"2022-06-15T16:06:05Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:14Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"f33afb065bcea72b766948ebbe5e5138493c8b06d5d86e3cdf4bea72c732c298","receipts":1}],"first_statement_date":"2022-06-15","related":[],"current":{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-1eaf00-a1","payload":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-1eaf00","subject_id":"gary-marcus","normalized_proposition":"Vigorous mechanisms should be enforced to ensure drivers using Level 2 driver-assist don't tune out.","kind":"normative","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"For example, we lack data on whether drivers use such systems as directed, and need to enforce vigorous mechanisms to make sure drivers don’t tune out.","locator":"paragraph 37","url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","date":"2022-06-15","document_title":"Will Tesla ever “Solve” Full Self Driving?","observed_at":"2026-10-29T00:05:14Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"6ab135b46717aba766bdbf01060d28f09af3fa5828be76db0bf1fd4863b6df90","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:56:19Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:56:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-will-tesla-ever-solve-full-self-driving-1eaf00-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-25b514","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-25b514-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-25b514#assessment-marcus-will-tesla-ever-solve-full-self-driving-25b514-a1"}],"claim":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-25b514","subject_id":"gary-marcus","revision":1,"content_hash":"cf170400b28230789d1cdfc2c0d8bb5dec80dc3f676cdb5b77b1d2e2ce332334","normalized_proposition":"If the whole US fleet used current self-driving software, incidents would rise with more vehicles and harder conditions, and likely jump radically without humans in the loop.","kind":"conditional_forecast","target":"driver-assist / self-driving software","conditions":[{"text":"if the entire US fleet used such software","type":"antecedent","from_source":true},{"text":"with more difficult conditions","type":"antecedent","from_source":true},{"text":"if humans were taken out of the loop","type":"antecedent","from_source":true}],"original_horizon_text":null,"topic_tags":["reliability","safety"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-will-tesla-ever-solve-full-self-driving-25b514-o1","statement_date":"2022-06-15","statement_date_precision":"day","excerpt":"Clearly, the number of incidents would go up, as the number of vehicles went up, and would also go up significantly as the systems were more regularly challenged with more difficult driving conditions (bad weather, more erratic pedestrians, etc), and likely jump radically if we took humans out of the loop altogether.","locator":"paragraph 27","document":{"url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","title":"Will Tesla ever “Solve” Full Self Driving?","publisher":"Marcus on AI","published_at":"2022-06-15T16:06:05Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:14Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"f33afb065bcea72b766948ebbe5e5138493c8b06d5d86e3cdf4bea72c732c298","receipts":1}],"first_statement_date":"2022-06-15","related":[],"current":{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-25b514-a1","payload":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-25b514","subject_id":"gary-marcus","normalized_proposition":"If the whole US fleet used current self-driving software, incidents would rise with more vehicles and harder conditions, and likely jump radically without humans in the loop.","kind":"conditional_forecast","conditions":["if the entire US fleet used such software","with more difficult conditions","if humans were taken out of the loop"],"original_horizon_text":null,"statement":{"excerpt":"Clearly, the number of incidents would go up, as the number of vehicles went up, and would also go up significantly as the systems were more regularly challenged with more difficult driving conditions (bad weather, more erratic pedestrians, etc), and likely jump radically if we took humans out of the loop altogether.","locator":"paragraph 27","url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","date":"2022-06-15","document_title":"Will Tesla ever “Solve” Full Self Driving?","observed_at":"2026-10-29T00:05:14Z"},"criteria":{"from_speaker":["if the entire US fleet used such software","with more difficult conditions","if humans were taken out of the loop"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found that evidence could not settle it: one found it about something no public evidence could show, such as a private intention or a counterfactual; the other found it stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_public_evidence","probability_language"]},"payload_hash":"3b22f0481dbe177e55a3639b2fe97375e5952cf094e525b5b85a2513c6a70659","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:56:19Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:56:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-will-tesla-ever-solve-full-self-driving-25b514-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-30e65b","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-30e65b-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-30e65b#assessment-marcus-will-tesla-ever-solve-full-self-driving-30e65b-a1"}],"claim":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-30e65b","subject_id":"gary-marcus","revision":2,"content_hash":"febc338d214748549c3d9ea828deabb3071bf3b3694cdbd40fab8c5fa23c5095","normalized_proposition":"Although he does not know whether it is good or bad, Tesla's roughly 0.75 ADAS incidents per day is enough to be concerning because it comes from a very limited trial of computer-assisted driving relative to the ambition of full self-driving.","kind":"normative","target":"Tesla Autopilot","conditions":[{"text":"he does not know whether the rate is good or bad","type":"qualifier","from_source":true},{"text":"from a very limited trial relative to the ambition of full self-driving","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["reliability","safety"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-will-tesla-ever-solve-full-self-driving-30e65b-o1","statement_date":"2022-06-15","statement_date_precision":"day","excerpt":"Is that good, or bad? I don’t know it, but it’s enough to be concerning. Why? Because .75 incidents a day comes from a very limited trial of computer-assisted driving relative to the ambition of full-self-driving.","locator":"paragraph 22","document":{"url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","title":"Will Tesla ever “Solve” Full Self Driving?","publisher":"Marcus on AI","published_at":"2022-06-15T16:06:05Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:14Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"f33afb065bcea72b766948ebbe5e5138493c8b06d5d86e3cdf4bea72c732c298","receipts":1}],"first_statement_date":"2022-06-15","related":[],"current":{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-30e65b-a1","payload":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-30e65b","subject_id":"gary-marcus","normalized_proposition":"Although he does not know whether it is good or bad, Tesla's roughly 0.75 ADAS incidents per day is enough to be concerning because it comes from a very limited trial of computer-assisted driving relative to the ambition of full self-driving.","kind":"normative","conditions":["he does not know whether the rate is good or bad","from a very limited trial relative to the ambition of full self-driving"],"original_horizon_text":null,"statement":{"excerpt":"Is that good, or bad? I don’t know it, but it’s enough to be concerning. Why? Because .75 incidents a day comes from a very limited trial of computer-assisted driving relative to the ambition of full-self-driving.","locator":"paragraph 22","url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","date":"2022-06-15","document_title":"Will Tesla ever “Solve” Full Self Driving?","observed_at":"2026-10-29T00:05:14Z"},"criteria":{"from_speaker":["he does not know whether the rate is good or bad","from a very limited trial relative to the ambition of full self-driving","0.75 incidents a day"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"c5c0e13c90e59528a0680bbe95564b69f703bbd3154556a022f2c904828e8102","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-will-tesla-ever-solve-full-self-driving-30e65b-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-5fcaec","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-5fcaec-a3","status":"current","url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-5fcaec#assessment-marcus-will-tesla-ever-solve-full-self-driving-5fcaec-a3"}],"claim":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-5fcaec","subject_id":"gary-marcus","revision":1,"content_hash":"7fe87da0e2509ba1602a079269f2f7de0e7c0b5f30d78e8da2f7797d81fff27a","normalized_proposition":"In the long run, driverless cars will save lives and give the blind, elderly, and disabled far more autonomy.","kind":"forecast","target":"driverless cars","conditions":[],"original_horizon_text":"in the long run","topic_tags":["safety","capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-will-tesla-ever-solve-full-self-driving-5fcaec-o1","statement_date":"2022-06-15","statement_date_precision":"day","excerpt":"Does that mean that society should give up on building driverless cars? Absolutely not. In the long run, they will save lives, and also give many people, such the blind, elderly, and disabled, have far more autonomy over their own lives.","locator":"paragraph 39","document":{"url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","title":"Will Tesla ever “Solve” Full Self Driving?","publisher":"Marcus on AI","published_at":"2022-06-15T16:06:05Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:14Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"f33afb065bcea72b766948ebbe5e5138493c8b06d5d86e3cdf4bea72c732c298","receipts":1}],"first_statement_date":"2022-06-15","related":[],"current":{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-5fcaec-a3","payload":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-5fcaec","subject_id":"gary-marcus","normalized_proposition":"In the long run, driverless cars will save lives and give the blind, elderly, and disabled far more autonomy.","kind":"forecast","conditions":[],"original_horizon_text":"in the long run","statement":{"excerpt":"Does that mean that society should give up on building driverless cars? Absolutely not. In the long run, they will save lives, and also give many people, such the blind, elderly, and disabled, have far more autonomy over their own lives.","locator":"paragraph 39","url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","date":"2022-06-15","document_title":"Will Tesla ever “Solve” Full Self Driving?","observed_at":"2026-10-29T00:05:14Z"},"criteria":{"from_speaker":["In the long run","they will save lives","many people","the blind, elderly, and disabled","far more autonomy over their own lives"],"our_interpretation":["In the long run supplies no calendar deadline.","Driverless cars includes services that transport passengers without a human driver; this benefits claim does not explicitly require Level 5.","Save lives means fewer deaths than would otherwise occur for comparable travel, including effects on other road users.","Far more autonomy means substantial, sustained gains in independent travel and access to activities for many people in the named groups.","Both the safety and independence components must be established to support the combined forecast."],"support_condition":"Credible real-world evidence establishes reduced fatalities attributable to driverless deployment and substantial, sustained independent-mobility gains for many blind, older, and disabled people.","contradiction_condition":"Decisive evidence establishes that the promised long-run benefits cannot occur. Present shortcomings or a limited deployment's failure do not alone contradict an undated forecast."},"axis":"forecast_outcome","outcome":"unresolved","horizon_state":"open_horizon","evaluation_as_of":"2026-10-28","rationale":"IIHS research released July 23, 2026 found lower crash involvement rates for the Waymo driverless operations studied during 2021–2024 [e1]. Waymo-authored research published May 2, 2025 reported lower injury-related crash rates over 56.7 million rider-only miles through January 2025 [e2]. These findings do not separately establish fewer fatalities attributable to deployment [e1, e2]. AFB's June 2026 report, with fieldwork ending October 15, 2025, recorded accessibility barriers and reported that only 36% of surveyed AV riders could afford rides whenever needed or wanted [e3]. It does not establish substantial, sustained independence gains for many blind, older and disabled people [e3]. The evidence therefore leaves the combined forecast unresolved: it establishes neither both promised benefits nor their long-run impossibility [e1, e2, e3].","limitations":["The safety findings apply to the studied Waymo operations and do not separately establish a fatality reduction [e1, e2].","The injury study is company-authored [e2].","The AFB findings report rider experiences and affordability, without establishing sustained independence gains at meaningful scale across the named populations [e3].","Present access barriers do not establish that long-run benefits cannot occur [e3]."],"evidence":[{"id":"e1","relation":"context","excerpt":"The results provide further evidence that Waymo’s current L4 vehicles have lower crash involvement rates than human drivers.","locator":"Abstract, Conclusions","url":"https://www.iihs.org/research-areas/bibliography/ref/2374","publisher":"Insurance Institute for Highway Safety","event_date":"2024","available_at":"2026-07-23","explanation":"The study of 2021–2024 records supports a safety benefit for Waymo's studied operations; the event date marks the final study year. Crash reductions do not independently establish the forecast's fatality and mobility components."},{"id":"e2","relation":"context","excerpt":"Cyclist, Motorcycle, Pedestrian, Secondary Crash, and Single Vehicle crashes were also statistically reduced for the Any-Injury-Reported outcome.","locator":"Abstract, results paragraph","url":"https://arxiv.org/abs/2505.01515","publisher":"arXiv","event_date":"2025-01","available_at":"2025-05-02","explanation":"This Waymo-authored study examines 56.7 million rider-only miles through January 2025 and reports favorable injury outcomes. It strengthens the safety case but does not separately establish reduced deaths or broad independence gains."},{"id":"e3","relation":"context","excerpt":"Regardless of disability status, only 36% of AV riders said they could afford to pay for an AV whenever they wanted or needed one.","locator":"AV Rider Experiences, AV Affordability","url":"https://www.afb.org/research-and-initiatives/ai-series/navigating-toward-ai","publisher":"American Foundation for the Blind","event_date":"2025-10-15","available_at":"2026-06-30","explanation":"The survey records enthusiasm alongside affordability and accessibility barriers. Fieldwork ended October 15, 2025; the report specifies June 2026 publication, so June 30 is a conservative availability bound rather than an asserted publication day."}],"what_would_resolve":"Credible exposure-adjusted evidence of reduced fatalities attributable to driverless deployment, including effects on other road users, plus substantial, sustained independence gains at meaningful scale among blind, older and disabled people. Mobility studies should measure independent travel, access, affordability and usability. Decisive evidence that the promised long-run benefits cannot occur would instead contradict the forecast.","retrospective":true,"plain_reading":true},"payload_hash":"2e48de908e4b18d6728032428da04110bd8744d7ea82b1c2a54f2bfa73c1eca9","policy_version":"2026-10-28.7","recorded_at":"2026-10-31T19:31:23Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-127663-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-will-tesla-ever-solve-full-self-driving-127663-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-127663-2x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"independent_measurement","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e3","source_type":"independent_measurement","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-31T19:31:23Z","next_review_due_at":null,"horizon_now":"open_horizon"},"event_ids":["marcus-will-tesla-ever-solve-full-self-driving-5fcaec-assessment-published-a3","marcus-will-tesla-ever-solve-full-self-driving-5fcaec-held-a2"],"last_substantive_change_at":"2026-10-31T22:29:51Z"}},{"url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-9ca231-2","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-9ca231-2-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-9ca231-2#assessment-marcus-will-tesla-ever-solve-full-self-driving-9ca231-2-a2"}],"claim":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-9ca231-2","subject_id":"gary-marcus","revision":1,"content_hash":"7055920cd830ff557c68df6db90a4fd0222c48ee9c7c53cf79622bf4ade14ecf","normalized_proposition":"Without a solution to the outlier problem, full self-driving will not be achieved.","kind":"conditional_forecast","target":"full self-driving","conditions":[{"text":"without a solution to the outlier problem","type":"antecedent","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities","reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-will-tesla-ever-solve-full-self-driving-9ca231-2-o1","statement_date":"2022-06-15","statement_date_precision":"day","excerpt":"As far as I can tell, nobody has a real solution to the outlier problem of coping with unusual circumstances; without it, full self driving will remain a pipe dream.","locator":"paragraph 34","document":{"url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","title":"Will Tesla ever “Solve” Full Self Driving?","publisher":"Marcus on AI","published_at":"2022-06-15T16:06:05Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:14Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"f33afb065bcea72b766948ebbe5e5138493c8b06d5d86e3cdf4bea72c732c298","receipts":1}],"first_statement_date":"2022-06-15","related":[],"current":{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-9ca231-2-a2","payload":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-9ca231-2","subject_id":"gary-marcus","normalized_proposition":"Without a solution to the outlier problem, full self-driving will not be achieved.","kind":"conditional_forecast","conditions":["without a solution to the outlier problem"],"original_horizon_text":null,"statement":{"excerpt":"As far as I can tell, nobody has a real solution to the outlier problem of coping with unusual circumstances; without it, full self driving will remain a pipe dream.","locator":"paragraph 34","url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","date":"2022-06-15","document_title":"Will Tesla ever “Solve” Full Self Driving?","observed_at":"2026-10-29T00:05:14Z"},"criteria":{"from_speaker":["without it","the outlier problem of coping with unusual circumstances","full self driving will remain a pipe dream"],"our_interpretation":["Assess the conditional forecast, rather than separately grading the preceding qualified observation.","In context, full self-driving means Level 5 operation across human-drivable roads and conditions.","A solution means reliable handling of unusual driving circumstances, not perfect avoidance of every conceivable accident.","The absence of Level 5 deployment does not establish either the antecedent or the claimed necessity."],"support_condition":"Evidence establishes that reliable handling of unusual circumstances is necessary for unrestricted autonomous driving, with the absence of such a solution independently established.","contradiction_condition":"A verified Level 5 system achieves unrestricted autonomy while independent evidence establishes that it has not solved reliable handling of unusual circumstances."},"axis":"forecast_outcome","outcome":"unresolved","horizon_state":"open_horizon","evaluation_as_of":"2026-10-28","rationale":"On July 23, 2026, IIHS reported that Level 5 vehicles did not yet exist and described Waymo's vehicles as Level 4, operating under specified conditions [e1]. The report does not independently establish whether reliable handling of unusual circumstances remains unsolved or whether that explains the absence of unrestricted autonomy [e1]. It supplies neither evidence establishing the claimed necessity nor a verified Level 5 counterexample with the outlier problem independently shown to remain unsolved [e1]. The conditional forecast remains unresolved on this evidence [e1].","limitations":["The evidence provides no independent definition or test establishing whether an outlier solution exists [e1].","Continued Level 5 non-achievement does not establish the conditional claim's necessity [e1].","The report describes the state on July 23, 2026; it does not verify the status through October 28, 2026 [e1]."],"evidence":[{"id":"e1","relation":"context","excerpt":"Level 5 vehicles, capable of driving without human input under all conditions, do not yet exist.","locator":"Paragraph explaining Level 3 and Level 5 systems","url":"https://www.iihs.org/news/detail/waymos-driverless-cars-crash-less-often-than-people","publisher":"Insurance Institute for Highway Safety","event_date":"2026-07-23","available_at":"2026-07-23","explanation":"IIHS reports no Level 5 achievement. This supplies no counterexample, but does not independently establish the absence of an outlier solution or prove that such absence caused non-achievement."}],"what_would_resolve":"Independent technical evidence that tests unusual-situation handling separately from Level 5 capability, establishes the absence of a solution, and demonstrates that such a solution is necessary for unrestricted autonomy. A dated, independently verified Level 5 achievement while the absence of an outlier solution is separately established would conclusively contradict the forecast.","retrospective":true,"plain_reading":true},"payload_hash":"8860d5c1ab92c4ce4eab3a13808dcb3bdb4541c31c0d7f4a248c181ca5c50bf2","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T16:49:38Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-7ea600-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-will-tesla-ever-solve-full-self-driving-7ea600-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-127663-3x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"independent_measurement","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T16:49:38Z","next_review_due_at":null,"horizon_now":"open_horizon"},"event_ids":["marcus-will-tesla-ever-solve-full-self-driving-9ca231-2-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-ae62e8","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-ae62e8-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-ae62e8#assessment-marcus-will-tesla-ever-solve-full-self-driving-ae62e8-a1"}],"claim":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-ae62e8","subject_id":"gary-marcus","revision":1,"content_hash":"341dfdfe1c84b2ee9fb7ca3665f86dadf98a91fff95ea6517913907796a6ddf5","normalized_proposition":"Tesla might not survive as a company if its massive bet on self-driving fails.","kind":"conditional_forecast","target":"Tesla","conditions":[{"text":"if its bet on self-driving fails","type":"antecedent","from_source":true}],"original_horizon_text":null,"topic_tags":["business-economics","capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-will-tesla-ever-solve-full-self-driving-ae62e8-o1","statement_date":"2022-06-15","statement_date_precision":"day","excerpt":"The best argument that Tesla might not survive if its massive bet on self-driving fails comes from Elon Musk himself.","locator":"paragraph 1","document":{"url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","title":"Will Tesla ever “Solve” Full Self Driving?","publisher":"Marcus on AI","published_at":"2022-06-15T16:06:05Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:14Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"f33afb065bcea72b766948ebbe5e5138493c8b06d5d86e3cdf4bea72c732c298","receipts":1}],"first_statement_date":"2022-06-15","related":[],"current":{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-ae62e8-a1","payload":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-ae62e8","subject_id":"gary-marcus","normalized_proposition":"Tesla might not survive as a company if its massive bet on self-driving fails.","kind":"conditional_forecast","conditions":["if its bet on self-driving fails"],"original_horizon_text":null,"statement":{"excerpt":"The best argument that Tesla might not survive if its massive bet on self-driving fails comes from Elon Musk himself.","locator":"paragraph 1","url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","date":"2022-06-15","document_title":"Will Tesla ever “Solve” Full Self Driving?","observed_at":"2026-10-29T00:05:14Z"},"criteria":{"from_speaker":["if its bet on self-driving fails"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"16b86600d3271849bb00430718f2153a4d2dbcb58466a5a43cd437eed403048c","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:56:19Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:56:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-will-tesla-ever-solve-full-self-driving-ae62e8-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-b923eb","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-b923eb-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-b923eb#assessment-marcus-will-tesla-ever-solve-full-self-driving-b923eb-a1"}],"claim":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-b923eb","subject_id":"gary-marcus","revision":1,"content_hash":"018c88a433da2cd9a55332da7f031bdab194ffdecb3baee6e7f9bcc91314cc10","normalized_proposition":"Level 5 self-driving that handles everything has not been achieved.","kind":"observation","target":"Level 5 self-driving","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-will-tesla-ever-solve-full-self-driving-b923eb-o1","statement_date":"2022-06-15","statement_date_precision":"day","excerpt":"Level 5 has to handle it all; everyday stopped vehicles, weird outliers and so forth. We’re just not there.","locator":"paragraph 36","document":{"url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","title":"Will Tesla ever “Solve” Full Self Driving?","publisher":"Marcus on AI","published_at":"2022-06-15T16:06:05Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:14Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"f33afb065bcea72b766948ebbe5e5138493c8b06d5d86e3cdf4bea72c732c298","receipts":1}],"first_statement_date":"2022-06-15","related":[],"current":{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-b923eb-a1","payload":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-b923eb","subject_id":"gary-marcus","normalized_proposition":"Level 5 self-driving that handles everything has not been achieved.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Level 5 has to handle it all; everyday stopped vehicles, weird outliers and so forth. We’re just not there.","locator":"paragraph 36","url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","date":"2022-06-15","document_title":"Will Tesla ever “Solve” Full Self Driving?","observed_at":"2026-10-29T00:05:14Z"},"criteria":{"from_speaker":["Level 5","handle it all","everyday stopped vehicles, weird outliers","We’re just not there"],"our_interpretation":["The plain reading is that publicly demonstrated Level 5 driving had not been achieved at publication.","Level 5 means driving across conditions that a human can navigate, without an operational-domain restriction or required human fallback. It does not mean immunity from every accident.","For the ongoing technology-status observation, use June 2022 up to the June 15 post. There is no regular quarterly reporting cycle for this capability."],"support_condition":"Contemporaneous evidence reports no achieved Level 5 system and distinguishes existing bounded driverless deployments from unrestricted capability.","contradiction_condition":"A system had demonstrably achieved unrestricted Level 5 capability by June 15, 2022."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Reporting on June 15 explicitly states that no Level 5 vehicles existed. The contemporaneous Cruise permit independently illustrates the distinction: driverless passenger service existed, but within substantial operating restrictions. That supports Marcus's specific Level 5 observation without denying the progress already made in bounded autonomy.","limitations":["The industry-wide finding relies on contemporaneous reporting; the primary permit corroborates a particular deployment.","The assessment concerns publicly demonstrated capability, not undisclosed prototypes.","The schema lacks a regulatory-disclosure source category; the CPUC item is classified as company_disclosure but was issued by a regulator."],"evidence":[{"id":"e1","relation":"supports","excerpt":"(Notably, there are no Level 5 vehicles in existence today.)","locator":"Paragraph explaining Levels 2–5, before the discussion of data limitations","url":"https://www.theverge.com/2022/6/15/23168088/nhtsa-adas-self-driving-crash-data-tesla","publisher":"The Verge","event_date":"2022-06-15","available_at":"2022-06-15","explanation":"The contemporaneous report directly addresses the claimed industry-wide state. Reputable reporting is used because no readable primary source establishing that complete industry-wide negative was obtained."},{"id":"e2","relation":"context","excerpt":"This includes, but is not limited to, expansions to the hours, geography, roadway types, speed range, or weather conditions of its operations.","locator":"Second substantive paragraph, describing changes requiring an updated Passenger Safety Plan","url":"https://www.cpuc.ca.gov/news-and-updates/all-news/cpuc-issues-first-driverless-autonomous-vehicle-passenger-service-deployment-permit","publisher":"California Public Utilities Commission","event_date":"2022-06-02","available_at":"2022-06-02","explanation":"The permit allowed Cruise driverless passenger service on selected San Francisco streets, at limited speeds and hours, with weather exclusions. It confirms meaningful driverless deployment while showing why that deployment did not establish Level 5 capability."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"ec14353bdf56a2536b5dc809e00dc2f0548d16effc0b4ddbf071080e4c56f9ec","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:56:44Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-7ea600-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-will-tesla-ever-solve-full-self-driving-7ea600-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-7ea600-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"news_report","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:56:44Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-will-tesla-ever-solve-full-self-driving-b923eb-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-bc7108","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-bc7108-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-bc7108#assessment-marcus-will-tesla-ever-solve-full-self-driving-bc7108-a1"}],"claim":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-bc7108","subject_id":"gary-marcus","revision":2,"content_hash":"d343a00f7c5183d8420425f93ea35dbe51f955f42b3e77ade536d163c523d548","normalized_proposition":"Much more data, such as miles driven, is needed if we are to assess in a truly scientific way how safe driver-assist and self-driving systems really are.","kind":"normative","target":"driver-assist safety data","conditions":[{"text":"if we are to assess safety in a truly scientific way","type":"qualifier","from_source":true},{"text":"such as miles driven","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["regulation-policy","safety"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-will-tesla-ever-solve-full-self-driving-bc7108-o1","statement_date":"2022-06-15","statement_date_precision":"day","excerpt":"It’s fantastic that NHTSA is now pushing manufacturers to release incident data, and but we need soooo much more data (eg about miles driven) if we are to look at the money question—how safe is all this, really—in a truly scientific way.","locator":"paragraph 17","document":{"url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","title":"Will Tesla ever “Solve” Full Self Driving?","publisher":"Marcus on AI","published_at":"2022-06-15T16:06:05Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:14Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"f33afb065bcea72b766948ebbe5e5138493c8b06d5d86e3cdf4bea72c732c298","receipts":1}],"first_statement_date":"2022-06-15","related":[],"current":{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-bc7108-a1","payload":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-bc7108","subject_id":"gary-marcus","normalized_proposition":"Much more data, such as miles driven, is needed if we are to assess in a truly scientific way how safe driver-assist and self-driving systems really are.","kind":"normative","conditions":["if we are to assess safety in a truly scientific way","such as miles driven"],"original_horizon_text":null,"statement":{"excerpt":"It’s fantastic that NHTSA is now pushing manufacturers to release incident data, and but we need soooo much more data (eg about miles driven) if we are to look at the money question—how safe is all this, really—in a truly scientific way.","locator":"paragraph 17","url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","date":"2022-06-15","document_title":"Will Tesla ever “Solve” Full Self Driving?","observed_at":"2026-10-29T00:05:14Z"},"criteria":{"from_speaker":["if we are to assess safety in a truly scientific way","such as miles driven"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"6e80b27fddb55c9aa9bd1a59709255666685f26bc3556371fc37d9de1e073c42","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-will-tesla-ever-solve-full-self-driving-bc7108-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-c0396b","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-c0396b-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-c0396b#assessment-marcus-will-tesla-ever-solve-full-self-driving-c0396b-a1"}],"claim":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-c0396b","subject_id":"gary-marcus","revision":1,"content_hash":"885d64a064c7f8dd72ff496e16cdf14df3f3db05ddb492a6afa7ffc11c989c98","normalized_proposition":"Driverless car experiments should be strictly regulated, and data transparency requirements should be vastly stricter.","kind":"normative","target":"driverless car experiments","conditions":[{"text":"especially when nonconsenting people are involved","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["regulation-policy","safety"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-will-tesla-ever-solve-full-self-driving-c0396b-o1","statement_date":"2022-06-15","statement_date_precision":"day","excerpt":"But our experiments in building them need to be strictly regulated, especially when nonconsenting human beings become unwilling participants. And our requirements on data transparency need to be vastly stricter.","locator":"paragraph 40","document":{"url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","title":"Will Tesla ever “Solve” Full Self Driving?","publisher":"Marcus on AI","published_at":"2022-06-15T16:06:05Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:14Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"f33afb065bcea72b766948ebbe5e5138493c8b06d5d86e3cdf4bea72c732c298","receipts":1}],"first_statement_date":"2022-06-15","related":[],"current":{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-c0396b-a1","payload":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-c0396b","subject_id":"gary-marcus","normalized_proposition":"Driverless car experiments should be strictly regulated, and data transparency requirements should be vastly stricter.","kind":"normative","conditions":["especially when nonconsenting people are involved"],"original_horizon_text":null,"statement":{"excerpt":"But our experiments in building them need to be strictly regulated, especially when nonconsenting human beings become unwilling participants. And our requirements on data transparency need to be vastly stricter.","locator":"paragraph 40","url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","date":"2022-06-15","document_title":"Will Tesla ever “Solve” Full Self Driving?","observed_at":"2026-10-29T00:05:14Z"},"criteria":{"from_speaker":["especially when nonconsenting people are involved"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"3246cf2865d435dac6d9543dd1386b41cef373e721edc02e7fda5f7bee8782f1","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:56:19Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:56:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-will-tesla-ever-solve-full-self-driving-c0396b-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-c509c9","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-c509c9-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-c509c9#assessment-marcus-will-tesla-ever-solve-full-self-driving-c509c9-a1"}],"claim":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-c509c9","subject_id":"gary-marcus","revision":1,"content_hash":"975564508035a4729f7cb148fb1373cc69e5e2026a8446e1b210db5ae72e71d7","normalized_proposition":"Society should not give up on building driverless cars.","kind":"normative","target":"driverless cars","conditions":[],"original_horizon_text":null,"topic_tags":["other"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-will-tesla-ever-solve-full-self-driving-c509c9-o1","statement_date":"2022-06-15","statement_date_precision":"day","excerpt":"Does that mean that society should give up on building driverless cars? Absolutely not.","locator":"paragraph 39","document":{"url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","title":"Will Tesla ever “Solve” Full Self Driving?","publisher":"Marcus on AI","published_at":"2022-06-15T16:06:05Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:14Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"f33afb065bcea72b766948ebbe5e5138493c8b06d5d86e3cdf4bea72c732c298","receipts":1}],"first_statement_date":"2022-06-15","related":[],"current":{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-c509c9-a1","payload":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-c509c9","subject_id":"gary-marcus","normalized_proposition":"Society should not give up on building driverless cars.","kind":"normative","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Does that mean that society should give up on building driverless cars? Absolutely not.","locator":"paragraph 39","url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","date":"2022-06-15","document_title":"Will Tesla ever “Solve” Full Self Driving?","observed_at":"2026-10-29T00:05:14Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"d000381cfab28fbcd2f6eeb49efce84fa90668b8b44baf05207cfe95cc0d28a2","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:56:19Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:56:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-will-tesla-ever-solve-full-self-driving-c509c9-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-ca8bc2","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-ca8bc2-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-ca8bc2#assessment-marcus-will-tesla-ever-solve-full-self-driving-ca8bc2-a1"}],"claim":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-ca8bc2","subject_id":"gary-marcus","revision":2,"content_hash":"79e3afd700eb7480fe1d6a9fbd67829194d46c2b414696a53ba9ea3a08f980d9","normalized_proposition":"The new NHTSA data should be a bit of a wakeup call to those who have gone long on Tesla thinking a Full Self Driving solution is in sight; the data aren't devastating, but they aren't so promising either, and they raise a lot of questions.","kind":"normative","target":"Tesla Full Self Driving","conditions":[{"text":"those who have gone long on Tesla","type":"scope","from_source":true},{"text":"if they think a Full Self Driving solution is in sight","type":"scope","from_source":true},{"text":"a bit of a wakeup call","type":"qualifier","from_source":true},{"text":"aren't so promising","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["reliability","capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-will-tesla-ever-solve-full-self-driving-ca8bc2-o1","statement_date":"2022-06-15","statement_date_precision":"day","excerpt":"This morning’s fresh data dump from NHTSA should be a bit of a wakeup call to those who have gone long on Tesla, if they are thinking that a solution to Full Self Driving is in sight. The data aren’t devastating, but they aren’t so promising either, and they raise of lot of questions.","locator":"paragraph 3","document":{"url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","title":"Will Tesla ever “Solve” Full Self Driving?","publisher":"Marcus on AI","published_at":"2022-06-15T16:06:05Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:14Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"f33afb065bcea72b766948ebbe5e5138493c8b06d5d86e3cdf4bea72c732c298","receipts":1}],"first_statement_date":"2022-06-15","related":[],"current":{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-ca8bc2-a1","payload":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-ca8bc2","subject_id":"gary-marcus","normalized_proposition":"The new NHTSA data should be a bit of a wakeup call to those who have gone long on Tesla thinking a Full Self Driving solution is in sight; the data aren't devastating, but they aren't so promising either, and they raise a lot of questions.","kind":"normative","conditions":["those who have gone long on Tesla","if they think a Full Self Driving solution is in sight","a bit of a wakeup call","aren't so promising"],"original_horizon_text":null,"statement":{"excerpt":"This morning’s fresh data dump from NHTSA should be a bit of a wakeup call to those who have gone long on Tesla, if they are thinking that a solution to Full Self Driving is in sight. The data aren’t devastating, but they aren’t so promising either, and they raise of lot of questions.","locator":"paragraph 3","url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","date":"2022-06-15","document_title":"Will Tesla ever “Solve” Full Self Driving?","observed_at":"2026-10-29T00:05:14Z"},"criteria":{"from_speaker":["those who have gone long on Tesla","if they think a Full Self Driving solution is in sight","a bit of a wakeup call","aren't so promising"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"c403dad5c389aa5f0670584f154f5d27e24d295aecc6324b6a6862a5235694d1","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-will-tesla-ever-solve-full-self-driving-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-will-tesla-ever-solve-full-self-driving-ca8bc2-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-de30d0","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-de30d0-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-will-tesla-ever-solve-full-self-driving-de30d0#assessment-marcus-will-tesla-ever-solve-full-self-driving-de30d0-a1"}],"claim":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-de30d0","subject_id":"gary-marcus","revision":1,"content_hash":"ef5ef1a76cc11e40743f5e62d0fb0cfa5350eae8654b2177c2102c9ee86d1889","normalized_proposition":"NHTSA data show even ordinary circumstances like avoiding stopped vehicles remain problematic for automated driving systems.","kind":"observation","target":"driver-assist and self-driving systems","conditions":[],"original_horizon_text":null,"topic_tags":["reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-will-tesla-ever-solve-full-self-driving-de30d0-o1","statement_date":"2022-06-15","statement_date_precision":"day","excerpt":"Indeed, the NHTSA report data show that even some pretty ordinary circumstances, like avoiding stopped vehicles, are continuing to prove problematic.","locator":"paragraph 35","document":{"url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","title":"Will Tesla ever “Solve” Full Self Driving?","publisher":"Marcus on AI","published_at":"2022-06-15T16:06:05Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:14Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"f33afb065bcea72b766948ebbe5e5138493c8b06d5d86e3cdf4bea72c732c298","receipts":1}],"first_statement_date":"2022-06-15","related":[],"current":{"assessment_id":"marcus-will-tesla-ever-solve-full-self-driving-de30d0-a1","payload":{"claim_id":"marcus-will-tesla-ever-solve-full-self-driving-de30d0","subject_id":"gary-marcus","normalized_proposition":"NHTSA data show even ordinary circumstances like avoiding stopped vehicles remain problematic for automated driving systems.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Indeed, the NHTSA report data show that even some pretty ordinary circumstances, like avoiding stopped vehicles, are continuing to prove problematic.","locator":"paragraph 35","url":"https://garymarcus.substack.com/p/will-tesla-ever-solve-full-self-driving","date":"2022-06-15","document_title":"Will Tesla ever “Solve” Full Self Driving?","observed_at":"2026-10-29T00:05:14Z"},"criteria":{"from_speaker":["the NHTSA report data","ordinary circumstances","avoiding stopped vehicles","continuing to prove problematic"],"our_interpretation":["The plain reading is that NHTSA documented continuing collisions involving stopped vehicles during automated or assisted operation.","Problematic does not specify a crash-rate threshold, establish fault in every incident, or imply that these systems perform worse than human drivers.","Use July 2021 through May 15, 2022, the reporting window associated with the initial June 2022 release.","Contemporaneous NHTSA investigation records can corroborate the underlying observation; distinguish them from the specific June 15 summary reports."],"support_condition":"NHTSA records document collisions within the reporting window in which vehicles operating with driving automation failed to avoid stopped vehicles.","contradiction_condition":"The relevant records show that the purported stopped-vehicle examples did not occur, involved no driving automation, or were solely cases of stationary automated vehicles being struck by others."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"NHTSA's contemporaneous investigation documents additional Autopilot collisions with stopped first-responder vehicles during the relevant window. That supports the modest observation that avoiding stopped vehicles remained problematic. It does not establish a failure rate, blame automation for every crash, or show that all automated-driving systems shared the same problem.","limitations":["The decisive source is NHTSA's contemporaneous investigation, not a verified extraction of the specific June 15 summary tables.","These are Level 2 Tesla incidents and cannot establish the performance of every ADS developer.","The investigation also examines driver supervision and human behavior."],"evidence":[{"id":"e1","relation":"supports","excerpt":"Three crashes (Orlando, Petaluma, Desert Center)\ninvolved first responder or crash attenuator trucks in the roadway and occurred after PE21-020 opened.","locator":"Detailed Summary, paragraph describing additional first-responder crashes; Incident List (EA22-002 Upgrade)","url":"https://static.nhtsa.gov/odi/inv/2022/INOA-EA22002-3184.PDF","publisher":"National Highway Traffic Safety Administration","event_date":"2022-01","available_at":"2022-06-09","explanation":"The surrounding summary identifies these as additional Autopilot collisions with stopped first-responder vehicles. Its incident list dates Orlando to August 2021, Petaluma to September 2021, and Desert Center to January 2022; the event date records the latest of these in-window incidents."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"034345a9584df6ea4965923e577fcc400f73facf957543f06ea14a8dd49dce99","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:55:12Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-de30d0-1x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-will-tesla-ever-solve-full-self-driving-de30d0-1x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-will-tesla-ever-solve-full-self-driving-de30d0-1x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"official_statistics","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:55:12Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-will-tesla-ever-solve-full-self-driving-de30d0-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-28eaa9","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-nonsense-on-stilts-28eaa9-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-28eaa9#assessment-marcus-nonsense-on-stilts-28eaa9-a1"}],"claim":{"claim_id":"marcus-nonsense-on-stilts-28eaa9","subject_id":"gary-marcus","revision":1,"content_hash":"61c98e5118bc0dd83a6ec50bd8f7dcfadfd5e092ac47625b1386eaff31420d6f","normalized_proposition":"Everything LaMDA says is bullshit with no real meaning behind it.","kind":"rhetoric","target":"LaMDA","conditions":[],"original_horizon_text":null,"topic_tags":["reliability","capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-nonsense-on-stilts-28eaa9-o1","statement_date":"2022-06-12","statement_date_precision":"day","excerpt":"In truth, literally everything that the system says is bullshit.","locator":"paragraph 63","document":{"url":"https://garymarcus.substack.com/p/nonsense-on-stilts","title":"Nonsense on Stilts","publisher":"Marcus on AI","published_at":"2022-06-12T20:43:01Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:16Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"2334304145eda517e503fe8bfeeb02f914b70633b820b315fe8b0990d5cdeed5","receipts":1}],"first_statement_date":"2022-06-12","related":[],"current":{"assessment_id":"marcus-nonsense-on-stilts-28eaa9-a1","payload":{"claim_id":"marcus-nonsense-on-stilts-28eaa9","subject_id":"gary-marcus","normalized_proposition":"Everything LaMDA says is bullshit with no real meaning behind it.","kind":"rhetoric","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"In truth, literally everything that the system says is bullshit.","locator":"paragraph 63","url":"https://garymarcus.substack.com/p/nonsense-on-stilts","date":"2022-06-12","document_title":"Nonsense on Stilts","observed_at":"2026-10-29T00:05:16Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this rhetoric that commits to nothing testable.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["rhetoric"]},"payload_hash":"bc2ead4bd5093674b29c1348536250db1980c40a55c96dab5174926f00562321","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-nonsense-on-stilts-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-nonsense-on-stilts-28eaa9-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-29cb82","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-nonsense-on-stilts-29cb82-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-29cb82#assessment-marcus-nonsense-on-stilts-29cb82-a1"}],"claim":{"claim_id":"marcus-nonsense-on-stilts-29cb82","subject_id":"gary-marcus","revision":1,"content_hash":"98b2b70c8c8c6bc2f8366bf647d1484e0e8ac8dedeb7fb34ce90df78cc317384","normalized_proposition":"Nearly all of the AI community considers the idea that LaMDA might be sentient ridiculous.","kind":"observation","target":"the AI community","conditions":[],"original_horizon_text":null,"topic_tags":["other"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-nonsense-on-stilts-29cb82-o1","statement_date":"2022-06-12","statement_date_precision":"day","excerpt":"We in the AI community have our differences, but pretty much all of find the notion that LaMDA might be sentient completely ridiculous.","locator":"paragraph 24","document":{"url":"https://garymarcus.substack.com/p/nonsense-on-stilts","title":"Nonsense on Stilts","publisher":"Marcus on AI","published_at":"2022-06-12T20:43:01Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:16Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"2334304145eda517e503fe8bfeeb02f914b70633b820b315fe8b0990d5cdeed5","receipts":1}],"first_statement_date":"2022-06-12","related":[],"current":{"assessment_id":"marcus-nonsense-on-stilts-29cb82-a1","payload":{"claim_id":"marcus-nonsense-on-stilts-29cb82","subject_id":"gary-marcus","normalized_proposition":"Nearly all of the AI community considers the idea that LaMDA might be sentient ridiculous.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"We in the AI community have our differences, but pretty much all of find the notion that LaMDA might be sentient completely ridiculous.","locator":"paragraph 24","url":"https://garymarcus.substack.com/p/nonsense-on-stilts","date":"2022-06-12","document_title":"Nonsense on Stilts","observed_at":"2026-10-29T00:05:16Z"},"criteria":{"from_speaker":["We in the AI community","pretty much all","LaMDA might be sentient","completely ridiculous"],"our_interpretation":["Read 'pretty much all' as an overwhelming field-wide consensus, allowing rare dissent, rather than literal unanimity.","The claim concerns researchers' attitudes toward LaMDA's possible sentience, not whether LaMDA actually was sentient.","Use June 1 through June 12, 2022, for the ongoing state during the controversy immediately surrounding publication.","A selection of prominent skeptics can support widespread skepticism without establishing the prevalence or strength of that attitude throughout the field."],"support_condition":"Representative evidence from the period establishes that an overwhelming share of the AI community dismissed LaMDA's possible sentience, rather than merely finding the available evidence insufficient.","contradiction_condition":"Representative evidence establishes substantial uncertainty or dissent in the AI community during the period. A few individual dissenters would not suffice."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Contemporaneous reporting strongly supports widespread expert skepticism and agreement among prominent public respondents. It does not establish that nearly the entire AI community dismissed even the possibility of LaMDA's sentience as ridiculous. The evidence supports a weaker statement, but does not settle the claim's breadth and strength.","limitations":["Public commentators are a self-selected group.","The ABC account describes views after the target period.","An informal reading of the claim as merely broad expert skepticism would receive stronger support."],"evidence":[{"id":"e1","relation":"context","excerpt":"almost every member","locator":"Opening discussion, paragraph beginning 'After the story’s publication'","url":"https://fortune.com/2022/06/13/google-ai-researchers-sentient-chatbot-claims-ridiculed-by-experts","publisher":"Fortune","event_date":"2022-06","available_at":"2022-06-13","explanation":"The report describes agreement among prominent researchers who publicly responded to the story. It provides evidence of visible consensus, but no sampling frame or measurement of the entire AI community."},{"id":"e2","relation":"context","excerpt":"expert consensus","locator":"Discussion following comments from David Chalmers and Toby Walsh","url":"https://www.abc.net.au/news/science/2022-06-15/google-ai-chatbot-not-sentient-how-do-we-know-intelligence/101150090","publisher":"Australian Broadcasting Corporation","event_date":"2022-06-15","available_at":"2022-06-15","explanation":"This independently supports the existence of broad expert skepticism shortly after publication. It neither measures near-unanimity nor establishes that almost everyone regarded the possibility as ridiculous by June 12."}],"what_would_resolve":"A representative survey or broad documented review of researcher attitudes during early June 2022 that distinguishes disbelief, uncertainty and outright dismissal.","retrospective":true,"plain_reading":true},"payload_hash":"217cc0f99b34e7846319dd878db1e6e047a80b7c5d5b1c4ae79f5448ce879251","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T21:49:13Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-29cb82-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-nonsense-on-stilts-29cb82-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-29cb82-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"news_report","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"news_report","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T21:49:13Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-nonsense-on-stilts-29cb82-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T22:52:30Z","probable":{"call":"probably_right","level":"reasonable","reasons":[{"provider":"anthropic","reason":"Fortune's contemporaneous report says almost every researcher who weighed in agreed LaMDA was not sentient, and ABC describes an expert consensus. Neither is a survey, but both point the same way."},{"provider":"openai","reason":"Fortune reports near-unanimous rejection among prominent researchers responding to LaMDA, and ABC independently describes an expert consensus against its consciousness. Together they support Marcus's ordinary consensus claim."}]}}},{"url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-48f130","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-nonsense-on-stilts-48f130-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-48f130#assessment-marcus-nonsense-on-stilts-48f130-a1"}],"claim":{"claim_id":"marcus-nonsense-on-stilts-48f130","subject_id":"gary-marcus","revision":1,"content_hash":"782c0905fff3163d70d0578187e75dcc946d84b2ed82f62ec6439c7b0b686905","normalized_proposition":"Fooling people into thinking a program is intelligent is not the same as building an intelligent program.","kind":"rhetoric","target":"chatbots","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-nonsense-on-stilts-48f130-o1","statement_date":"2022-06-12","statement_date_precision":"day","excerpt":"Fooling people into thinking a program is intelligent is just not the same as building programs that actually are intelligent.","locator":"paragraph 53","document":{"url":"https://garymarcus.substack.com/p/nonsense-on-stilts","title":"Nonsense on Stilts","publisher":"Marcus on AI","published_at":"2022-06-12T20:43:01Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:16Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"2334304145eda517e503fe8bfeeb02f914b70633b820b315fe8b0990d5cdeed5","receipts":1}],"first_statement_date":"2022-06-12","related":[],"current":{"assessment_id":"marcus-nonsense-on-stilts-48f130-a1","payload":{"claim_id":"marcus-nonsense-on-stilts-48f130","subject_id":"gary-marcus","normalized_proposition":"Fooling people into thinking a program is intelligent is not the same as building an intelligent program.","kind":"rhetoric","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Fooling people into thinking a program is intelligent is just not the same as building programs that actually are intelligent.","locator":"paragraph 53","url":"https://garymarcus.substack.com/p/nonsense-on-stilts","date":"2022-06-12","document_title":"Nonsense on Stilts","observed_at":"2026-10-29T00:05:16Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this rhetoric that commits to nothing testable.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["rhetoric"]},"payload_hash":"957547c42cdb04639f10fb3ce94976ee156ddff98a41cb53de5b3c502240b672","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-nonsense-on-stilts-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-nonsense-on-stilts-48f130-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-4f4543","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-nonsense-on-stilts-4f4543-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-4f4543#assessment-marcus-nonsense-on-stilts-4f4543-a1"}],"claim":{"claim_id":"marcus-nonsense-on-stilts-4f4543","subject_id":"gary-marcus","revision":2,"content_hash":"e0ac36af4b4fb30d47e4571234c2bf42a612b94cc2271eee4762d4a55ebd5fba","normalized_proposition":"The AI community wishes the Turing Test would go away, and its members all know that beating the test is not meaningful.","kind":"observation","target":"the Turing Test","conditions":[{"text":"the AI community, as opposed to the public","type":"scope","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-nonsense-on-stilts-4f4543-o1","statement_date":"2022-06-12","statement_date_precision":"day","excerpt":"The public knows the test of course, but the AI community wishes it would go away; we all know that beating that test isn’t meaningful.","locator":"paragraph 42","document":{"url":"https://garymarcus.substack.com/p/nonsense-on-stilts","title":"Nonsense on Stilts","publisher":"Marcus on AI","published_at":"2022-06-12T20:43:01Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:16Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"2334304145eda517e503fe8bfeeb02f914b70633b820b315fe8b0990d5cdeed5","receipts":1}],"first_statement_date":"2022-06-12","related":[],"current":{"assessment_id":"marcus-nonsense-on-stilts-4f4543-a1","payload":{"claim_id":"marcus-nonsense-on-stilts-4f4543","subject_id":"gary-marcus","normalized_proposition":"The AI community wishes the Turing Test would go away, and its members all know that beating the test is not meaningful.","kind":"observation","conditions":["the AI community, as opposed to the public"],"original_horizon_text":null,"statement":{"excerpt":"The public knows the test of course, but the AI community wishes it would go away; we all know that beating that test isn’t meaningful.","locator":"paragraph 42","url":"https://garymarcus.substack.com/p/nonsense-on-stilts","date":"2022-06-12","document_title":"Nonsense on Stilts","observed_at":"2026-10-29T00:05:16Z"},"criteria":{"from_speaker":["the AI community wishes it would go away","we all know","beating that test isn’t meaningful"],"our_interpretation":["The test is the conversational Turing Test discussed in the surrounding passage.","Read 'we all' as a claim of broad field consensus, rather than a literal claim about every individual.","Meaningful refers to passing the test as a measure of AI progress. The claim also asserts a community preference to abandon it.","Use June 1 through June 12, 2022, for the ongoing community attitude described at publication.","The reference to the AI community rather than the public specifies the population; it is not a conditional forecast."],"support_condition":"Representative evidence establishes both broad rejection of passing the conversational Turing Test as meaningful evidence of AI progress and a broad preference to abandon it.","contradiction_condition":"Evidence establishes a substantial active constituency in the AI community that assigns the test meaningful value or wants to retain it. Isolated disagreement does not alone disprove the ordinary consensus reading."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The verified scholarly record contains both criticism and continuing endorsement of the Turing Test. It does not establish a June 2022 field-wide wish to abandon it, or the prevalence of views about its significance. A literal unanimity reading faces documented dissent, but that alone does not settle the fairer reading as a broad-consensus claim.","limitations":["The usable scholarly overview predates the target period.","Criticism of the test's limitations is not equivalent to believing it has no value.","Later Turing-test research was not used to infer earlier community attitudes."],"evidence":[{"id":"e1","relation":"context","excerpt":"continue to endorse The Turing Test.","locator":"Section 5, discussion immediately before subsection 5.1","url":"https://plato.stanford.edu/archives/spr2022/entries/turing-test","publisher":"Stanford Encyclopedia of Philosophy","event_date":"2021-10-04","available_at":"2021-10-04","explanation":"The overview documents both criticism and continuing defense, including work by Neufeld and Finnestad. It cautions against literal unanimity but does not measure the size of either constituency in June 2022."}],"what_would_resolve":"Representative contemporaneous evidence covering both perceived benchmark value and willingness to retain or abandon the test.","retrospective":true,"plain_reading":true},"payload_hash":"b0dfb5e85e020a21fcaf936ed20566b78862075c96ee7f4c46595388ca81d501","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T21:49:13Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-29cb82-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-nonsense-on-stilts-29cb82-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-29cb82-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"secondary_summary","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T21:49:13Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-nonsense-on-stilts-4f4543-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T22:52:30Z","probable":{"call":"probably_wrong","level":"speculative","reasons":[{"provider":"openai","reason":"The Stanford Encyclopedia's prepublication overview describes deeply divided views on the Turing Test's value and documents continuing defenses. That points against Marcus's claim of community agreement that passing it is meaningless."}]}}},{"url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-52d256","standing":"current","assessment_label":"Contradicted","versions":[{"assessment_id":"marcus-nonsense-on-stilts-52d256-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-52d256#assessment-marcus-nonsense-on-stilts-52d256-a1"}],"claim":{"claim_id":"marcus-nonsense-on-stilts-52d256","subject_id":"gary-marcus","revision":1,"content_hash":"db38f4a89e562f399d9a189834fb8ac83c55522c7a6e65ccb2eadc39c360f792","normalized_proposition":"Software like LaMDA does not connect to the world; it is just autocomplete predicting which words fit a context.","kind":"observation","target":"LaMDA","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-nonsense-on-stilts-52d256-o1","statement_date":"2022-06-12","statement_date_precision":"day","excerpt":"Software like LaMDA simply doesn’t; it doesn’t even try to connect to the world at large, it just tries to be the best version of autocomplete it can be, by predicting what words best fit a given context.","locator":"paragraph 17","document":{"url":"https://garymarcus.substack.com/p/nonsense-on-stilts","title":"Nonsense on Stilts","publisher":"Marcus on AI","published_at":"2022-06-12T20:43:01Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:16Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"2334304145eda517e503fe8bfeeb02f914b70633b820b315fe8b0990d5cdeed5","receipts":1}],"first_statement_date":"2022-06-12","related":[],"current":{"assessment_id":"marcus-nonsense-on-stilts-52d256-a1","payload":{"claim_id":"marcus-nonsense-on-stilts-52d256","subject_id":"gary-marcus","normalized_proposition":"Software like LaMDA does not connect to the world; it is just autocomplete predicting which words fit a context.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Software like LaMDA simply doesn’t; it doesn’t even try to connect to the world at large, it just tries to be the best version of autocomplete it can be, by predicting what words best fit a given context.","locator":"paragraph 17","url":"https://garymarcus.substack.com/p/nonsense-on-stilts","date":"2022-06-12","document_title":"Nonsense on Stilts","observed_at":"2026-10-29T00:05:16Z"},"criteria":{"from_speaker":["Software like LaMDA","doesn’t even try to connect to the world at large","just tries to be the best version of autocomplete","predicting what words best fit a given context"],"our_interpretation":["Assess the publicly described LaMDA dialogue system, including its grounding tools, rather than only its pretrained base model.","The ordinary technical reading of connecting to the world includes consulting external information to ground factual answers. It does not require consciousness or human understanding.","For this ongoing technical observation, use the current-year documentation window from the beginning of 2022 through the post date. There is no standard reporting quarter for this system.","Next-token prediction can coexist with external retrieval. Establishing the former alone does not establish the claim's restrictive 'just' and 'doesn’t even try'."],"support_condition":"Technical documentation establishes that LaMDA only generates contextual text and contains no mechanism or training objective for consulting external information to ground its answers.","contradiction_condition":"Technical documentation establishes that LaMDA explicitly attempts factual grounding through external knowledge sources or tools."},"axis":"historical_observation","outcome":"contradicted","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Google's published LaMDA system explicitly learned to consult external knowledge and tools to ground responses. That contradicts 'doesn’t even try' under the ordinary technical reading. Next-token prediction remains part of its operation, but does not establish that the complete system is only autocomplete. This finding does not establish understanding or sentience.","limitations":["The result depends on reading external grounding as a connection to the world.","The evidence concerns the documented research system, not the configuration of every private demonstration."],"evidence":[{"id":"e1","relation":"contradicts","excerpt":"external knowledge resources and tools.","locator":"Groundedness discussion, immediately before 'The toolset (TS)'","url":"https://arxiv.org/html/2201.08239v3","publisher":"Google Research authors / arXiv","event_date":"2022-01","available_at":"2022-02-10","explanation":"The paper describes retrieval, calculator and translation tools, and training that routes queries to those tools during inference. This directly contradicts the assertion that LaMDA does not even attempt external grounding."},{"id":"e2","relation":"context","excerpt":"pre-training and fine-tuning.","locator":"Section 'LaMDA Pre-Training', opening sentence","url":"https://research.google/blog/lamda-towards-safe-grounded-and-high-quality-dialog-models-for-everything","publisher":"Google Research","event_date":"2022-01-21","available_at":"2022-01-21","explanation":"The surrounding sections confirm next-token pretraining, but also describe safety classification, response ranking and external factual grounding. Marcus's description captures the generative training objective without capturing the complete system."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"1c36a094d50d5abd2407bc8bc7866ee906fdff2d863f76cf2abed77641dc00a0","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T21:49:13Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-29cb82-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-nonsense-on-stilts-29cb82-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-29cb82-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T21:49:13Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-nonsense-on-stilts-52d256-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T22:52:30Z"}},{"url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-75d580","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-nonsense-on-stilts-75d580-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-75d580#assessment-marcus-nonsense-on-stilts-75d580-a1"}],"claim":{"claim_id":"marcus-nonsense-on-stilts-75d580","subject_id":"gary-marcus","revision":1,"content_hash":"400c5146b2ae373a1f71628e892bcab74abe1d8f8c7e23a4b2a01ba05d2b70a5","normalized_proposition":"Neither LaMDA nor similar large language models such as GPT-3 are remotely intelligent.","kind":"observation","target":"LaMDA and similar language models","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-nonsense-on-stilts-75d580-o1","statement_date":"2022-06-12","statement_date_precision":"day","excerpt":"Neither LaMDA nor any of its cousins (GPT-3) are remotely intelligent.","locator":"paragraph 12","document":{"url":"https://garymarcus.substack.com/p/nonsense-on-stilts","title":"Nonsense on Stilts","publisher":"Marcus on AI","published_at":"2022-06-12T20:43:01Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:16Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"2334304145eda517e503fe8bfeeb02f914b70633b820b315fe8b0990d5cdeed5","receipts":1}],"first_statement_date":"2022-06-12","related":[],"current":{"assessment_id":"marcus-nonsense-on-stilts-75d580-a1","payload":{"claim_id":"marcus-nonsense-on-stilts-75d580","subject_id":"gary-marcus","normalized_proposition":"Neither LaMDA nor similar large language models such as GPT-3 are remotely intelligent.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Neither LaMDA nor any of its cousins (GPT-3) are remotely intelligent.","locator":"paragraph 12","url":"https://garymarcus.substack.com/p/nonsense-on-stilts","date":"2022-06-12","document_title":"Nonsense on Stilts","observed_at":"2026-10-29T00:05:16Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"7f5f4dc6237019968f99b336421d268f62b1be8dba8a8eec3fc4027111d9815d","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-nonsense-on-stilts-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-nonsense-on-stilts-75d580-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-87d145","standing":"current","assessment_label":"Contradicted","versions":[{"assessment_id":"marcus-nonsense-on-stilts-87d145-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-87d145#assessment-marcus-nonsense-on-stilts-87d145-a1"}],"claim":{"claim_id":"marcus-nonsense-on-stilts-87d145","subject_id":"gary-marcus","revision":1,"content_hash":"ebe92f60fda0097fae7fc43f7fdeb4e01b4e8b4aec220bf47b14a5c55b142bbd","normalized_proposition":"The Eugene Goostman chatbot, celebrated in 2014, made no lasting contribution to AI.","kind":"observation","target":"Eugene Goostman","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-nonsense-on-stilts-87d145-o1","statement_date":"2022-06-12","statement_date_precision":"day","excerpt":"But 8 years later I doubt most people (even in AI) have ever even heard of his program, outside of my mentioning it here. It made zero lasting contribution to AI.","locator":"paragraph 52","document":{"url":"https://garymarcus.substack.com/p/nonsense-on-stilts","title":"Nonsense on Stilts","publisher":"Marcus on AI","published_at":"2022-06-12T20:43:01Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:16Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"2334304145eda517e503fe8bfeeb02f914b70633b820b315fe8b0990d5cdeed5","receipts":1}],"first_statement_date":"2022-06-12","related":[],"current":{"assessment_id":"marcus-nonsense-on-stilts-87d145-a1","payload":{"claim_id":"marcus-nonsense-on-stilts-87d145","subject_id":"gary-marcus","normalized_proposition":"The Eugene Goostman chatbot, celebrated in 2014, made no lasting contribution to AI.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"But 8 years later I doubt most people (even in AI) have ever even heard of his program, outside of my mentioning it here. It made zero lasting contribution to AI.","locator":"paragraph 52","url":"https://garymarcus.substack.com/p/nonsense-on-stilts","date":"2022-06-12","document_title":"Nonsense on Stilts","observed_at":"2026-10-29T00:05:16Z"},"criteria":{"from_speaker":["8 years later","his program","It made zero lasting contribution to AI"],"our_interpretation":["The program is Eugene Goostman, and the assessment concerns its legacy from 2014 through June 12, 2022.","Assess the firm contribution claim. The preceding 'I doubt' statement about name recognition is a separate hedged aside.","An ordinary lasting contribution includes substantive reuse of its technology in later AI systems or enduring research advances attributable to it. It need not transform the entire field.","Publicity, museum display and citations alone do not establish a lasting technical contribution."],"support_condition":"Credible historical evidence establishes that the system left no enduring technical or research contribution during the period; absence of search hits alone is insufficient.","contradiction_condition":"Documented subsequent adoption of Eugene Goostman's code or methods, or a substantive enduring research contribution, establishes a nonzero legacy by June 12, 2022."},"axis":"historical_observation","outcome":"contradicted","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"A participant in the Emelya robot project explicitly traces its natural-language analysis system to Eugene Goostman. Her retrospective account dates development and product use to years before Marcus's post. This is a concrete technical legacy beyond publicity or citations, contradicting 'zero lasting contribution' under the ordinary reading that includes applied AI systems. It does not establish a major scientific breakthrough.","limitations":["The decisive evidence is a retrospective participant account published in 2023 about events in 2016–2018.","The original-language quotation is Russian; its meaning is explained in English.","Technical reuse is documented, but the amount of inherited code and the size of its contribution are not.","A narrower interpretation limited to foundational scientific advances would remain unresolved."],"evidence":[{"id":"e1","relation":"contradicts","excerpt":"переработанная система анализа естественного языка","locator":"Discussion of Eugene Goostman preceding the author's dialogue-research questions; release timing in the home-testing discussion","url":"https://habr.com/ru/companies/oleg-bunin/articles/704558","publisher":"Habr / Ontico, account by project participant Anna Degteva","event_date":"2017-05","available_at":"2023-01-17","explanation":"The author's statement says Emelya was based on a reworked natural-language analysis system created for Eugene Goostman. Her account dates development to 2016, release to May 2017 and further user research to 2018, documenting a technical legacy beyond the 2014 competition."},{"id":"e2","relation":"context","excerpt":"an improvement on their predecessor.","locator":"Abstract of 'Can machines talk? Comparison of Eliza with modern dialogue systems'","url":"https://portalrecerca.uab.cat/en/publications/can-machines-talk-comparison-of-eliza-with-modern-dialogue-system","publisher":"Computers in Human Behavior / Universitat Autònoma de Barcelona repository","event_date":"2016-05-01","available_at":"2016-05-01","explanation":"The published study compares Eugene Goostman and four other systems with ELIZA. It establishes substantive research use, but the group-level finding alone does not establish a lasting contribution unique to Eugene Goostman."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"15ac9380e1c8ac2d6ff50b7ee3042cf3b7dd97ccf5c32437adf569c20e9ece51","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T21:49:13Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-29cb82-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-nonsense-on-stilts-29cb82-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-29cb82-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"company_disclosure","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T21:49:13Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-nonsense-on-stilts-87d145-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T22:52:30Z"}},{"url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-96e570","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-nonsense-on-stilts-96e570-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-96e570#assessment-marcus-nonsense-on-stilts-96e570-a1"}],"claim":{"claim_id":"marcus-nonsense-on-stilts-96e570","subject_id":"gary-marcus","revision":1,"content_hash":"2e11944c0e61a19e46916dd854c56f2eb65df92eec7f5ff24f4a351bdfbe2e81","normalized_proposition":"LaMDA-style language models only match statistical patterns of human language, and their output has no real meaning.","kind":"observation","target":"LaMDA and similar language models","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-nonsense-on-stilts-96e570-o1","statement_date":"2022-06-12","statement_date_precision":"day","excerpt":"All they do is match patterns, draw from massive statistical databases of human language. The patterns might be cool, but language these systems utter doesn’t actually mean anything at all.","locator":"paragraph 12","document":{"url":"https://garymarcus.substack.com/p/nonsense-on-stilts","title":"Nonsense on Stilts","publisher":"Marcus on AI","published_at":"2022-06-12T20:43:01Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:16Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"2334304145eda517e503fe8bfeeb02f914b70633b820b315fe8b0990d5cdeed5","receipts":1}],"first_statement_date":"2022-06-12","related":[],"current":{"assessment_id":"marcus-nonsense-on-stilts-96e570-a1","payload":{"claim_id":"marcus-nonsense-on-stilts-96e570","subject_id":"gary-marcus","normalized_proposition":"LaMDA-style language models only match statistical patterns of human language, and their output has no real meaning.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"All they do is match patterns, draw from massive statistical databases of human language. The patterns might be cool, but language these systems utter doesn’t actually mean anything at all.","locator":"paragraph 12","url":"https://garymarcus.substack.com/p/nonsense-on-stilts","date":"2022-06-12","document_title":"Nonsense on Stilts","observed_at":"2026-10-29T00:05:16Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"d68bf8b573dd4ca7c9c0b1aaef145421c4669311ecdd27b66d5c8488bc61c385","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-nonsense-on-stilts-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-nonsense-on-stilts-96e570-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-9d49da","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-nonsense-on-stilts-9d49da-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-9d49da#assessment-marcus-nonsense-on-stilts-9d49da-a1"}],"claim":{"claim_id":"marcus-nonsense-on-stilts-9d49da","subject_id":"gary-marcus","revision":1,"content_hash":"f884d833881e32978069763d58809db4741cd5434953dac940f1b289a8ad8850","normalized_proposition":"No AI system that anyone knew how to build in 2022 is sentient.","kind":"observation","target":"AI systems buildable in 2022","conditions":[{"text":"anything anyone in 2022 knows how to build","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-nonsense-on-stilts-9d49da-o1","statement_date":"2022-06-12","statement_date_precision":"day","excerpt":"But there is no absolutely no reason whatever for us to waste time wondering whether anything anyone in 2022 knows how to build is sentient. It is not.","locator":"paragraph 65","document":{"url":"https://garymarcus.substack.com/p/nonsense-on-stilts","title":"Nonsense on Stilts","publisher":"Marcus on AI","published_at":"2022-06-12T20:43:01Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:16Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"2334304145eda517e503fe8bfeeb02f914b70633b820b315fe8b0990d5cdeed5","receipts":1}],"first_statement_date":"2022-06-12","related":[],"current":{"assessment_id":"marcus-nonsense-on-stilts-9d49da-a1","payload":{"claim_id":"marcus-nonsense-on-stilts-9d49da","subject_id":"gary-marcus","normalized_proposition":"No AI system that anyone knew how to build in 2022 is sentient.","kind":"observation","conditions":["anything anyone in 2022 knows how to build"],"original_horizon_text":null,"statement":{"excerpt":"But there is no absolutely no reason whatever for us to waste time wondering whether anything anyone in 2022 knows how to build is sentient. It is not.","locator":"paragraph 65","url":"https://garymarcus.substack.com/p/nonsense-on-stilts","date":"2022-06-12","document_title":"Nonsense on Stilts","observed_at":"2026-10-29T00:05:16Z"},"criteria":{"from_speaker":["anything anyone in 2022 knows how to build"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"3e5b0458178798cd01b1d91723233f3da67ab0447850f479913321fa6c3ccd44","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-nonsense-on-stilts-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-nonsense-on-stilts-9d49da-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-a29178","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-nonsense-on-stilts-a29178-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-a29178#assessment-marcus-nonsense-on-stilts-a29178-a1"}],"claim":{"claim_id":"marcus-nonsense-on-stilts-a29178","subject_id":"gary-marcus","revision":1,"content_hash":"21f597d7f787cdd32879351864a09e496bf50bc6a564c591053ae9a252fb19cf","normalized_proposition":"Marcus doubts that LaMDA and similar language models will play any important role in the future of AI.","kind":"forecast","target":"LaMDA and similar language models","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities","agi-timelines"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-nonsense-on-stilts-a29178-o1","statement_date":"2022-06-12","statement_date_precision":"day","excerpt":"None of the software in either of those systems has survived in modern efforts at “artificial general intelligence”, and I am not sure that LaMDA and its cousins will play any important role in the future of AI, either.","locator":"paragraph 15","document":{"url":"https://garymarcus.substack.com/p/nonsense-on-stilts","title":"Nonsense on Stilts","publisher":"Marcus on AI","published_at":"2022-06-12T20:43:01Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:16Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"2334304145eda517e503fe8bfeeb02f914b70633b820b315fe8b0990d5cdeed5","receipts":1}],"first_statement_date":"2022-06-12","related":[],"current":{"assessment_id":"marcus-nonsense-on-stilts-a29178-a1","payload":{"claim_id":"marcus-nonsense-on-stilts-a29178","subject_id":"gary-marcus","normalized_proposition":"Marcus doubts that LaMDA and similar language models will play any important role in the future of AI.","kind":"forecast","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"None of the software in either of those systems has survived in modern efforts at “artificial general intelligence”, and I am not sure that LaMDA and its cousins will play any important role in the future of AI, either.","locator":"paragraph 15","url":"https://garymarcus.substack.com/p/nonsense-on-stilts","date":"2022-06-12","document_title":"Nonsense on Stilts","observed_at":"2026-10-29T00:05:16Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"46617e58a945e322cce7c85322c42b3217a273df635be408b204201d5ee7596e","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-nonsense-on-stilts-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-nonsense-on-stilts-a29178-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-cd38ec","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-nonsense-on-stilts-cd38ec-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-cd38ec#assessment-marcus-nonsense-on-stilts-cd38ec-a1"}],"claim":{"claim_id":"marcus-nonsense-on-stilts-cd38ec","subject_id":"gary-marcus","revision":1,"content_hash":"7fb3a537f22c4d64ab703a0feff479c4c2efe4f31ada536cdf5428a8352bf7fb","normalized_proposition":"Language models like LaMDA string words together without any coherent understanding of the world.","kind":"observation","target":"LaMDA and similar language models","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-nonsense-on-stilts-cd38ec-o1","statement_date":"2022-06-12","statement_date_precision":"day","excerpt":"What these systems do, no more and no less, is to put together sequences of words, but without any coherent understanding of the world behind them, like foreign language Scrabble players who use English words as point-scoring tools, without any clue about what that mean.","locator":"paragraph 15","document":{"url":"https://garymarcus.substack.com/p/nonsense-on-stilts","title":"Nonsense on Stilts","publisher":"Marcus on AI","published_at":"2022-06-12T20:43:01Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:16Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"2334304145eda517e503fe8bfeeb02f914b70633b820b315fe8b0990d5cdeed5","receipts":1}],"first_statement_date":"2022-06-12","related":[],"current":{"assessment_id":"marcus-nonsense-on-stilts-cd38ec-a1","payload":{"claim_id":"marcus-nonsense-on-stilts-cd38ec","subject_id":"gary-marcus","normalized_proposition":"Language models like LaMDA string words together without any coherent understanding of the world.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"What these systems do, no more and no less, is to put together sequences of words, but without any coherent understanding of the world behind them, like foreign language Scrabble players who use English words as point-scoring tools, without any clue about what that mean.","locator":"paragraph 15","url":"https://garymarcus.substack.com/p/nonsense-on-stilts","date":"2022-06-12","document_title":"Nonsense on Stilts","observed_at":"2026-10-29T00:05:16Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"8b3f132ffe417fd6ac5f1b4b19749fc7e078fbec3eeeaea981026a673d8b54dd","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-nonsense-on-stilts-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-nonsense-on-stilts-cd38ec-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-d6fb09","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-nonsense-on-stilts-d6fb09-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-nonsense-on-stilts-d6fb09#assessment-marcus-nonsense-on-stilts-d6fb09-a1"}],"claim":{"claim_id":"marcus-nonsense-on-stilts-d6fb09","subject_id":"gary-marcus","revision":1,"content_hash":"1d12ae363190138f49fab7b148bbda6d22b83b4359365bd2cdec9c7491f63b6a","normalized_proposition":"LaMDA is not sentient.","kind":"observation","target":"LaMDA","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-nonsense-on-stilts-d6fb09-o1","statement_date":"2022-06-12","statement_date_precision":"day","excerpt":"To be sentient is to be aware of yourself in the world; LaMDA simply isn’t.","locator":"paragraph 15","document":{"url":"https://garymarcus.substack.com/p/nonsense-on-stilts","title":"Nonsense on Stilts","publisher":"Marcus on AI","published_at":"2022-06-12T20:43:01Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:16Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"2334304145eda517e503fe8bfeeb02f914b70633b820b315fe8b0990d5cdeed5","receipts":1}],"first_statement_date":"2022-06-12","related":[],"current":{"assessment_id":"marcus-nonsense-on-stilts-d6fb09-a1","payload":{"claim_id":"marcus-nonsense-on-stilts-d6fb09","subject_id":"gary-marcus","normalized_proposition":"LaMDA is not sentient.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"To be sentient is to be aware of yourself in the world; LaMDA simply isn’t.","locator":"paragraph 15","url":"https://garymarcus.substack.com/p/nonsense-on-stilts","date":"2022-06-12","document_title":"Nonsense on Stilts","observed_at":"2026-10-29T00:05:16Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"f6e27cfa583d609404cbc347911778647b39cbefb6cbf4a0a8841233fa4d01dd","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-nonsense-on-stilts-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-nonsense-on-stilts-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-nonsense-on-stilts-d6fb09-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-0f7b5b","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-0f7b5b-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-0f7b5b#assessment-marcus-does-ai-really-need-a-paradigm-shift-0f7b5b-a1"}],"claim":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-0f7b5b","subject_id":"gary-marcus","revision":1,"content_hash":"26d5b4e95304bd6bb83855310f69be588b86eecca504d04d4b2b078e766f021a","normalized_proposition":"The odds that scaling plus CNNs, RNNs and Transformers alone reach AGI with reliable reasoning are below 20%.","kind":"forecast","target":"scaling with CNNs + RNNs + Transformers alone","conditions":[{"text":"with scaling and CNNs + RNNs + Transformers alone","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["agi-timelines","scaling","reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-does-ai-really-need-a-paradigm-shift-0f7b5b-o1","statement_date":"2022-06-11","statement_date_precision":"day","excerpt":"In my view, no way; I think the actual odds are less than 20%.","locator":"paragraph 88","document":{"url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","title":"Does AI really need a paradigm shift?","publisher":"Marcus on AI","published_at":"2022-06-11T17:32:54Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:17Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"9716f559ab7b333c80f9fb7209db58ce8a0fde42f3f553914472aa7b9ff74804","receipts":1}],"first_statement_date":"2022-06-11","related":[],"current":{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-0f7b5b-a1","payload":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-0f7b5b","subject_id":"gary-marcus","normalized_proposition":"The odds that scaling plus CNNs, RNNs and Transformers alone reach AGI with reliable reasoning are below 20%.","kind":"forecast","conditions":["with scaling and CNNs + RNNs + Transformers alone"],"original_horizon_text":null,"statement":{"excerpt":"In my view, no way; I think the actual odds are less than 20%.","locator":"paragraph 88","url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","date":"2022-06-11","document_title":"Does AI really need a paradigm shift?","observed_at":"2026-10-29T00:05:17Z"},"criteria":{"from_speaker":["with scaling and CNNs + RNNs + Transformers alone","less than 20%"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"0e11040e2216a976aab6c26fd5577a7516ddc00dd5b42240ed48b279c1ec618b","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:10:54Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:10:54Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-does-ai-really-need-a-paradigm-shift-0f7b5b-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-13d459","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-13d459-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-13d459#assessment-marcus-does-ai-really-need-a-paradigm-shift-13d459-a1"}],"claim":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-13d459","subject_id":"gary-marcus","revision":1,"content_hash":"f7a19f77639cc3e1abf8602862c6c8d05606acb6e635b2f71e4d18b922bd5bfd","normalized_proposition":"Scale alone is not likely to be enough, and new discoveries (paradigm shifts) are needed, a point on which Marcus and LeCun agree.","kind":"forecast","target":"scaling alone","conditions":[],"original_horizon_text":null,"topic_tags":["scaling","agi-timelines"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-does-ai-really-need-a-paradigm-shift-13d459-o1","statement_date":"2022-06-11","statement_date_precision":"day","excerpt":"While there’s no denying that the two of us have often clashed, on this point—the point about scale alone not likely to be enough, and about the need for some new discovery (i.e., paradigm shifts)—we are actually in complete agreement.","locator":"paragraph 39","document":{"url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","title":"Does AI really need a paradigm shift?","publisher":"Marcus on AI","published_at":"2022-06-11T17:32:54Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:17Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"9716f559ab7b333c80f9fb7209db58ce8a0fde42f3f553914472aa7b9ff74804","receipts":1}],"first_statement_date":"2022-06-11","related":[],"current":{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-13d459-a1","payload":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-13d459","subject_id":"gary-marcus","normalized_proposition":"Scale alone is not likely to be enough, and new discoveries (paradigm shifts) are needed, a point on which Marcus and LeCun agree.","kind":"forecast","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"While there’s no denying that the two of us have often clashed, on this point—the point about scale alone not likely to be enough, and about the need for some new discovery (i.e., paradigm shifts)—we are actually in complete agreement.","locator":"paragraph 39","url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","date":"2022-06-11","document_title":"Does AI really need a paradigm shift?","observed_at":"2026-10-29T00:05:17Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"a29152d3651f97a85b4879ea3cac5dbdf60806852abd96dda10c31e384d4b8f0","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:10:54Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:10:54Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-does-ai-really-need-a-paradigm-shift-13d459-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-228a31","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-228a31-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-228a31#assessment-marcus-does-ai-really-need-a-paradigm-shift-228a31-a1"}],"claim":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-228a31","subject_id":"gary-marcus","revision":1,"content_hash":"f742401c5e54797666e2fcae9a840db44bc6149f45ef83b437c323def9da5b16","normalized_proposition":"GPT-2 is a clear sign that it is time to consider investing in approaches other than scaling.","kind":"normative","target":"AI research community","conditions":[{"text":"in light of massive data and compute poured into GPT-2","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["scaling"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-does-ai-really-need-a-paradigm-shift-228a31-o1","statement_date":"2022-06-11","statement_date_precision":"day","excerpt":"GPT-2 is both a triumph for empiricism, and, in light of the massive resources of data and computation that have been poured into them, a clear sign that it is time to consider investing in different approaches.","locator":"paragraph 14","document":{"url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","title":"Does AI really need a paradigm shift?","publisher":"Marcus on AI","published_at":"2022-06-11T17:32:54Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:17Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"9716f559ab7b333c80f9fb7209db58ce8a0fde42f3f553914472aa7b9ff74804","receipts":1}],"first_statement_date":"2022-06-11","related":[],"current":{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-228a31-a1","payload":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-228a31","subject_id":"gary-marcus","normalized_proposition":"GPT-2 is a clear sign that it is time to consider investing in approaches other than scaling.","kind":"normative","conditions":["in light of massive data and compute poured into GPT-2"],"original_horizon_text":null,"statement":{"excerpt":"GPT-2 is both a triumph for empiricism, and, in light of the massive resources of data and computation that have been poured into them, a clear sign that it is time to consider investing in different approaches.","locator":"paragraph 14","url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","date":"2022-06-11","document_title":"Does AI really need a paradigm shift?","observed_at":"2026-10-29T00:05:17Z"},"criteria":{"from_speaker":["in light of massive data and compute poured into GPT-2"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"d8676130488517eb8c18a90fd4063455f2f87bcdeadb3739ff64038c77db9ea9","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:10:54Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:10:54Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-does-ai-really-need-a-paradigm-shift-228a31-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-282b4f","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-282b4f-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-282b4f#assessment-marcus-does-ai-really-need-a-paradigm-shift-282b4f-a2"}],"claim":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-282b4f","subject_id":"gary-marcus","revision":1,"content_hash":"3d54c3b1893bf6fb4682ad44405436e19727438df015fe93a0e62c83fd04b2ec","normalized_proposition":"BIG-bench shows many capabilities improve with scale but not all, which bears out Marcus's 'Deep learning is hitting a wall' essay.","kind":"observation","target":"large language models","conditions":[{"text":"per BIG-bench findings","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["scaling","capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-does-ai-really-need-a-paradigm-shift-282b4f-o1","statement_date":"2022-06-11","statement_date_precision":"day","excerpt":"The basic finding was: loads of things scale, but not at all ( as foreseen both by Kurenkov in his essay last year and in my much lampooned but basically accurate essay, Deep learning is hitting a wall).","locator":"paragraph 36","document":{"url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","title":"Does AI really need a paradigm shift?","publisher":"Marcus on AI","published_at":"2022-06-11T17:32:54Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:17Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"9716f559ab7b333c80f9fb7209db58ce8a0fde42f3f553914472aa7b9ff74804","receipts":1}],"first_statement_date":"2022-06-11","related":[],"current":{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-282b4f-a2","payload":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-282b4f","subject_id":"gary-marcus","normalized_proposition":"BIG-bench shows many capabilities improve with scale but not all, which bears out Marcus's 'Deep learning is hitting a wall' essay.","kind":"observation","conditions":["per BIG-bench findings"],"original_horizon_text":null,"statement":{"excerpt":"The basic finding was: loads of things scale, but not at all ( as foreseen both by Kurenkov in his essay last year and in my much lampooned but basically accurate essay, Deep learning is hitting a wall).","locator":"paragraph 36","url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","date":"2022-06-11","document_title":"Does AI really need a paradigm shift?","observed_at":"2026-10-29T00:05:17Z"},"criteria":{"from_speaker":["The basic finding was","loads of things scale, but not at all","as foreseen","Deep learning is hitting a wall"],"our_interpretation":["The surrounding sentence about progress on some measures but not others makes not all the intended reading of the apparent typo not at all.","The test concerns the original June 2022 BIG-bench paper and its evaluated model sizes.","The reference to the earlier essay is assessed as corroboration of its limited contention that improvement was uneven across measures. It is not treated as proof of every argument in that essay or of a permanent scaling barrier."],"support_condition":"The original BIG-bench report finds broad improvements with model scale alongside tasks or measures that do not improve over the tested range.","contradiction_condition":"The original report finds improvement on all tested measures, or fails to report the claimed coexistence of improving and non-improving performance."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The June 9, 2022 BIG-bench report explicitly states that some tasks improve with scale while others show no improvement over the tested range [e1]. This supports the limited claim of uneven measured progress across tasks [e1].","limitations":["The finding does not establish a permanent scaling barrier or what happens beyond the tested model sizes [e1].","It corroborates uneven measured progress, not every argument in Marcus's earlier essay [e1]."],"evidence":[{"id":"e1","relation":"supports","excerpt":"no improvement with scale","locator":"Section 3.4, Linearity and breakthroughness: categorizing scaling behaviors","url":"https://arxiv.org/html/2206.04615v1","publisher":"Srivastava et al.; arXiv","event_date":"2022-06-09","available_at":"2022-06-09","explanation":"The original report directly states the coexistence Marcus describes. Sections 3.4.2 and 6.1 also explain that metric choice can conceal gradual progress and that some limitations may change with larger models or improved data."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"f051d99d80625bcad62b8e4e9877719d72baa2d4ae52941b8833cd4080f4b566","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T07:10:59Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-282b4f-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-does-ai-really-need-a-paradigm-shift-282b4f-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-282b4f-4x-final-anthropic-2"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T07:10:59Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-does-ai-really-need-a-paradigm-shift-282b4f-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-3bb4b4","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-3bb4b4-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-3bb4b4#assessment-marcus-does-ai-really-need-a-paradigm-shift-3bb4b4-a2"}],"claim":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-3bb4b4","subject_id":"gary-marcus","revision":2,"content_hash":"5fcab7de7e80d33cc24a33337be910016d082a952e32feec30559cf732ddb475","normalized_proposition":"Whenever AGI arrives, an explicit type/token distinction will be crucial to it.","kind":"conditional_forecast","target":"explicit type/token distinction in AGI","conditions":[{"text":"whenever AGI comes","type":"antecedent","from_source":true}],"original_horizon_text":null,"topic_tags":["agi-timelines","capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-does-ai-really-need-a-paradigm-shift-3bb4b4-o1","statement_date":"2022-06-11","statement_date_precision":"day","excerpt":"👉an explicit type/token distinction will be crucial","locator":"prediction 4 of 4 (paragraph 58)","document":{"url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","title":"Does AI really need a paradigm shift?","publisher":"Marcus on AI","published_at":"2022-06-11T17:32:54Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:17Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"9716f559ab7b333c80f9fb7209db58ce8a0fde42f3f553914472aa7b9ff74804","receipts":1}],"first_statement_date":"2022-06-11","related":[],"current":{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-3bb4b4-a2","payload":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-3bb4b4","subject_id":"gary-marcus","normalized_proposition":"Whenever AGI arrives, an explicit type/token distinction will be crucial to it.","kind":"conditional_forecast","conditions":["whenever AGI comes"],"original_horizon_text":null,"statement":{"excerpt":"👉an explicit type/token distinction will be crucial","locator":"prediction 4 of 4 (paragraph 58)","url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","date":"2022-06-11","document_title":"Does AI really need a paradigm shift?","observed_at":"2026-10-29T00:05:17Z"},"criteria":{"from_speaker":["whenever AGI comes","an explicit type/token distinction","will be crucial","tied not to time but to architecture"],"our_interpretation":["Treat arrival as a demonstrated system with broadly accepted human-level general cognitive capability. Broad acceptance is an evaluator safeguard.","Read type/token distinction as an explicit separation between a general kind or category and particular instances of it.","Language-model tokenization alone does not establish this distinction in the system's reasoning.","Read crucial as central dependence supported by component tests or equivalent evidence.","Test the first broadly accepted AGI; dependence in that system would support the arrival prediction without proving universal necessity."],"support_condition":"The AGI condition is established, and technical evidence shows an explicit kind-versus-instance representation whose removal substantially undermines the arriving system's general capability.","contradiction_condition":"The AGI condition is established, and complete architectural evidence or controlled tests show that the system achieves general capability without an explicit type/token distinction."},"axis":"forecast_outcome","outcome":"unresolved","horizon_state":"open_horizon","evaluation_as_of":"2026-10-28","rationale":"OpenAI's September 3, 2026 announcement reports GPT-6 Astra's capabilities [e1]. It does not establish broadly accepted AGI or show whether general capability depends on an explicit distinction between kinds and individual instances [e1]. This evidence therefore leaves the architectural prediction unresolved [e1].","limitations":["The announcement provides neither independent confirmation of AGI nor tests of dependence on an explicit type/token distinction [e1].","Its lack of qualifying architectural evidence does not establish that the distinction is absent [e1]."],"evidence":[{"id":"e1","relation":"context","excerpt":"We’re introducing GPT‑6 Astra","locator":"Introduction, first paragraph after update notices","url":"https://openai.com/index/gpt-6-astra","publisher":"OpenAI","event_date":"2026-09-03","available_at":"2026-09-03","explanation":"This is a developer's capability claim for a candidate system. It does not independently establish AGI, disclose an explicit kind-versus-instance mechanism, or show whether such a mechanism is crucial."}],"what_would_resolve":"A dated, broadly accepted AGI demonstration, with public architectural evidence and controlled tests or equivalent evidence showing central dependence on an explicit kind-versus-instance distinction, or successful general capability without it.","retrospective":true,"plain_reading":true},"payload_hash":"ae4d518d9c10654b8b600b544fd5e02029542403f94f25854a762ec3d7a036ca","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T07:10:59Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-282b4f-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-does-ai-really-need-a-paradigm-shift-282b4f-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-282b4f-4x-final-anthropic-2"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T07:10:59Z","next_review_due_at":null,"horizon_now":"open_horizon"},"event_ids":["marcus-does-ai-really-need-a-paradigm-shift-3bb4b4-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-410d51","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-410d51-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-410d51#assessment-marcus-does-ai-really-need-a-paradigm-shift-410d51-a1"}],"claim":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-410d51","subject_id":"gary-marcus","revision":1,"content_hash":"939c6f78f5c5fa50b90aef1478a21b3025583f618bf28295a843b3539a781fef","normalized_proposition":"The failures of large-scale systems, after a heavily funded test of the scaling hypothesis, are evidence that the field should seriously consider alternatives to scaling.","kind":"normative","target":"AI research community","conditions":[{"text":"given failures of large scale systems","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["scaling"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-does-ai-really-need-a-paradigm-shift-410d51-o1","statement_date":"2022-06-11","statement_date_precision":"day","excerpt":"I say that we have given the scaling hypothesis a really good look (with a larger budget than all but a handful of projects in history), and as such the failures of large scale systems are evidence (not proof) that we ought to seriously consider alternatives, e.g., here:","locator":"paragraph 12","document":{"url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","title":"Does AI really need a paradigm shift?","publisher":"Marcus on AI","published_at":"2022-06-11T17:32:54Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:17Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"9716f559ab7b333c80f9fb7209db58ce8a0fde42f3f553914472aa7b9ff74804","receipts":1}],"first_statement_date":"2022-06-11","related":[],"current":{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-410d51-a1","payload":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-410d51","subject_id":"gary-marcus","normalized_proposition":"The failures of large-scale systems, after a heavily funded test of the scaling hypothesis, are evidence that the field should seriously consider alternatives to scaling.","kind":"normative","conditions":["given failures of large scale systems"],"original_horizon_text":null,"statement":{"excerpt":"I say that we have given the scaling hypothesis a really good look (with a larger budget than all but a handful of projects in history), and as such the failures of large scale systems are evidence (not proof) that we ought to seriously consider alternatives, e.g., here:","locator":"paragraph 12","url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","date":"2022-06-11","document_title":"Does AI really need a paradigm shift?","observed_at":"2026-10-29T00:05:17Z"},"criteria":{"from_speaker":["given failures of large scale systems"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"9dd4241990aecfff50d983bd3f72251d708baf99efcc302c2ef97f1e3dd69b16","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:10:54Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:10:54Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-does-ai-really-need-a-paradigm-shift-410d51-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-7639f9","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-7639f9-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-7639f9#assessment-marcus-does-ai-really-need-a-paradigm-shift-7639f9-a2"}],"claim":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-7639f9","subject_id":"gary-marcus","revision":2,"content_hash":"2897d3c9ba5955523fb4e16c4ebbf88cead89bd62112b51eb6e71a1293916fe1","normalized_proposition":"Whenever AGI arrives, explicit cognitive models will be crucial to it.","kind":"conditional_forecast","target":"explicit cognitive models in AGI","conditions":[{"text":"whenever AGI comes","type":"antecedent","from_source":true}],"original_horizon_text":null,"topic_tags":["agi-timelines","capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-does-ai-really-need-a-paradigm-shift-7639f9-o1","statement_date":"2022-06-11","statement_date_precision":"day","excerpt":"👉explicit cognitive models will be crucial","locator":"prediction 2 of 4 (paragraph 56)","document":{"url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","title":"Does AI really need a paradigm shift?","publisher":"Marcus on AI","published_at":"2022-06-11T17:32:54Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:17Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"9716f559ab7b333c80f9fb7209db58ce8a0fde42f3f553914472aa7b9ff74804","receipts":1}],"first_statement_date":"2022-06-11","related":[],"current":{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-7639f9-a2","payload":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-7639f9","subject_id":"gary-marcus","normalized_proposition":"Whenever AGI arrives, explicit cognitive models will be crucial to it.","kind":"conditional_forecast","conditions":["whenever AGI comes"],"original_horizon_text":null,"statement":{"excerpt":"👉explicit cognitive models will be crucial","locator":"prediction 2 of 4 (paragraph 56)","url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","date":"2022-06-11","document_title":"Does AI really need a paradigm shift?","observed_at":"2026-10-29T00:05:17Z"},"criteria":{"from_speaker":["whenever AGI comes","explicit cognitive models","will be crucial","tied not to time but to architecture"],"our_interpretation":["Treat arrival as a demonstrated system with broadly accepted human-level general cognitive capability. Broad acceptance is an evaluator safeguard.","Read explicit cognitive models as identifiable representations of entities, relations, and states that the system uses to understand situations and reason about them.","A researcher's model of the system's cognition does not itself qualify; the representations must operate within the system.","Read crucial as central dependence demonstrated through component tests or equivalent evidence.","Test the architecture of the first broadly accepted AGI; support for that case would not prove necessity for every possible AGI."],"support_condition":"The AGI condition is established, and public architectural and component evidence shows that explicit cognitive models are central to the arriving system's general capability.","contradiction_condition":"The AGI condition is established, and complete architectural evidence or controlled tests show that the system attains and retains general capability without explicit cognitive models."},"axis":"forecast_outcome","outcome":"unresolved","horizon_state":"open_horizon","evaluation_as_of":"2026-10-28","rationale":"IBM's July 14, 2026 update states that there is no academic consensus on exactly what qualifies as AGI [e1]. This background source establishes neither a qualifying AGI arrival nor dependence on explicit cognitive models [e1]. It therefore leaves the architectural prediction unresolved [e1].","limitations":["The July background summary does not establish AGI's status through October 28, 2026 [e1].","It provides no inspection or component tests of explicit cognitive models within a qualifying system [e1]."],"evidence":[{"id":"e1","relation":"context","excerpt":"no consensus","locator":"What is artificial general intelligence (AGI)?, paragraph 2","url":"https://www.ibm.com/think/topics/artificial-general-intelligence","publisher":"IBM","event_date":"2026-07-14","available_at":"2026-07-14","explanation":"This explains why AGI's arrival cannot be inferred from the label alone. It is background, not an independent capability assessment or evidence about cognitive-model necessity."}],"what_would_resolve":"A dated, broadly accepted AGI demonstration, with public documentation identifying internal cognitive models and controlled tests or equivalent evidence showing central dependence on them, or successful general capability without them.","retrospective":true,"plain_reading":true},"payload_hash":"0c45a66cc443060e08fb7dae3be898b8e0e3438e6eff7452d0eb6c494d013bae","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T07:10:59Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-282b4f-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-does-ai-really-need-a-paradigm-shift-282b4f-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-282b4f-4x-final-anthropic-2"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"secondary_summary","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T07:10:59Z","next_review_due_at":null,"horizon_now":"open_horizon"},"event_ids":["marcus-does-ai-really-need-a-paradigm-shift-7639f9-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-b3d895","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-b3d895-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-b3d895#assessment-marcus-does-ai-really-need-a-paradigm-shift-b3d895-a1"}],"claim":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-b3d895","subject_id":"gary-marcus","revision":1,"content_hash":"ed0c3467774a5f2e6895aec28ed65a782ee9f52916ad7f516feb64913a10eff5","normalized_proposition":"A paradigm shift may be needed, but it could take a different form than the neurosymbolic one Marcus advocates.","kind":"forecast","target":"AI paradigm shift","conditions":[],"original_horizon_text":null,"topic_tags":["agi-timelines"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-does-ai-really-need-a-paradigm-shift-b3d895-o1","statement_date":"2022-06-11","statement_date_precision":"day","excerpt":"That said, I, of course, could be right about foreseeing the need for a paradigm shift, but wrong about what that paradigm shift turns out to be.","locator":"paragraph 70","document":{"url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","title":"Does AI really need a paradigm shift?","publisher":"Marcus on AI","published_at":"2022-06-11T17:32:54Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:17Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"9716f559ab7b333c80f9fb7209db58ce8a0fde42f3f553914472aa7b9ff74804","receipts":1}],"first_statement_date":"2022-06-11","related":[],"current":{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-b3d895-a1","payload":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-b3d895","subject_id":"gary-marcus","normalized_proposition":"A paradigm shift may be needed, but it could take a different form than the neurosymbolic one Marcus advocates.","kind":"forecast","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"That said, I, of course, could be right about foreseeing the need for a paradigm shift, but wrong about what that paradigm shift turns out to be.","locator":"paragraph 70","url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","date":"2022-06-11","document_title":"Does AI really need a paradigm shift?","observed_at":"2026-10-29T00:05:17Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"83c9af24cc8b5913c86c956b45dffaaf783f0ba381e3296f55a85e0bcc52befa","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:10:54Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:10:54Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-does-ai-really-need-a-paradigm-shift-b3d895-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-b5bb62","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-b5bb62-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-b5bb62#assessment-marcus-does-ai-really-need-a-paradigm-shift-b5bb62-a1"}],"claim":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-b5bb62","subject_id":"gary-marcus","revision":2,"content_hash":"9b1af26fbee594258341ef3b3541f25d18429b697d8d5d423e5f06da27a0a65d","normalized_proposition":"The BIG-bench paper looked at scaling on many measures and found progress on some but not others, and the measures that did not scale in it are what Marcus calls the 'wall'.","kind":"observation","target":"deep learning scaling","conditions":[{"text":"in the BIG-bench paper's findings","type":"scope","from_source":true},{"text":"progress on some but not others","type":"scope","from_source":true}],"original_horizon_text":null,"topic_tags":["scaling"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-does-ai-really-need-a-paradigm-shift-b5bb62-o1","statement_date":"2022-06-11","statement_date_precision":"day","excerpt":"The massive paper looks at scaling on many measures, and sees progress on some– but not others. The stuff that isn’t scaling is the “wall”.","locator":"paragraph 37","document":{"url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","title":"Does AI really need a paradigm shift?","publisher":"Marcus on AI","published_at":"2022-06-11T17:32:54Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:17Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"9716f559ab7b333c80f9fb7209db58ce8a0fde42f3f553914472aa7b9ff74804","receipts":1}],"first_statement_date":"2022-06-11","related":[],"current":{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-b5bb62-a1","payload":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-b5bb62","subject_id":"gary-marcus","normalized_proposition":"The BIG-bench paper looked at scaling on many measures and found progress on some but not others, and the measures that did not scale in it are what Marcus calls the 'wall'.","kind":"observation","conditions":["in the BIG-bench paper's findings","progress on some but not others"],"original_horizon_text":null,"statement":{"excerpt":"The massive paper looks at scaling on many measures, and sees progress on some– but not others. The stuff that isn’t scaling is the “wall”.","locator":"paragraph 37","url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","date":"2022-06-11","document_title":"Does AI really need a paradigm shift?","observed_at":"2026-10-29T00:05:17Z"},"criteria":{"from_speaker":["The massive paper looks at scaling on many measures","sees progress on some– but not others","The stuff that isn’t scaling is the “wall”"],"our_interpretation":["The paper is the original June 2022 BIG-bench report.","Progress means improvement in the reported task metrics across evaluated model sizes.","Marcus explicitly labels the non-improving measures the wall. In this sentence, that label does not independently establish that they can never improve."],"support_condition":"The original paper evaluates many tasks and reports improving scores on some while other scores remain flat over the tested scales.","contradiction_condition":"The original paper does not report that pattern, or every evaluated task measure improves with scale."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The original BIG-bench paper explicitly reports the pattern Marcus describes. His use of wall names the measures that did not improve in those experiments. The evidence supports that historical description, while leaving future scalability open.","limitations":["The finding applies to the evaluated models, scales and metrics.","Some metrics can conceal gradual progress.","This assessment does not establish an unavoidable or permanent wall."],"evidence":[{"id":"e1","relation":"supports","excerpt":"no improvement with scale","locator":"Section 3.4, Linearity and breakthroughness: categorizing scaling behaviors","url":"https://arxiv.org/html/2206.04615v1","publisher":"Srivastava et al.; arXiv","event_date":"2022-06-09","available_at":"2022-06-09","explanation":"The report directly confirms improving and non-improving task scores across evaluated scales. Its discussion of smoother metrics limits what those flat scores imply about underlying capabilities."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"9ea28de6b0fb618045a79207f884521e98139abbb5cf68f799abf803d23f0ff7","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T07:00:46Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-828976-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-does-ai-really-need-a-paradigm-shift-828976-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-828976-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T07:00:46Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-does-ai-really-need-a-paradigm-shift-b5bb62-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-b9155e","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-b9155e-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-b9155e#assessment-marcus-does-ai-really-need-a-paradigm-shift-b9155e-a1"}],"claim":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-b9155e","subject_id":"gary-marcus","revision":2,"content_hash":"bfa4ccb1f7b626caaf8d6e36459b5a6a96899826ae9d93be2ab07dad683e652a","normalized_proposition":"AI research needs new approaches rather than scale alone.","kind":"normative","target":"AI research","conditions":[],"original_horizon_text":null,"topic_tags":["scaling"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-does-ai-really-need-a-paradigm-shift-b9155e-o1","statement_date":"2022-06-11","statement_date_precision":"day","excerpt":"That’s why we need new approaches, exactly as I have been arguing.","locator":"paragraph 38","document":{"url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","title":"Does AI really need a paradigm shift?","publisher":"Marcus on AI","published_at":"2022-06-11T17:32:54Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:17Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"9716f559ab7b333c80f9fb7209db58ce8a0fde42f3f553914472aa7b9ff74804","receipts":1}],"first_statement_date":"2022-06-11","related":[],"current":{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-b9155e-a1","payload":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-b9155e","subject_id":"gary-marcus","normalized_proposition":"AI research needs new approaches rather than scale alone.","kind":"normative","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"That’s why we need new approaches, exactly as I have been arguing.","locator":"paragraph 38","url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","date":"2022-06-11","document_title":"Does AI really need a paradigm shift?","observed_at":"2026-10-29T00:05:17Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"a1c6963f262cf9eb28fd65e7236e0d2bc4a11fa701bba5837bb2a35f120f920c","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-does-ai-really-need-a-paradigm-shift-b9155e-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-c220b8","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-c220b8-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-c220b8#assessment-marcus-does-ai-really-need-a-paradigm-shift-c220b8-a2"}],"claim":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-c220b8","subject_id":"gary-marcus","revision":2,"content_hash":"81906cbc94ac0f2f17899904a1ae209698926a1f24641bc272f103300ad4068e","normalized_proposition":"Whenever AGI arrives, operations over variables, including storing, retrieving and comparing values, will be crucial to it.","kind":"conditional_forecast","target":"operations over variables in AGI","conditions":[{"text":"whenever AGI comes","type":"antecedent","from_source":true}],"original_horizon_text":null,"topic_tags":["agi-timelines","capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-does-ai-really-need-a-paradigm-shift-c220b8-o1","statement_date":"2022-06-11","statement_date_precision":"day","excerpt":"👉operations over variables (including storing, retrieving and comparing values) will be crucial","locator":"prediction 3 of 4 (paragraph 57)","document":{"url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","title":"Does AI really need a paradigm shift?","publisher":"Marcus on AI","published_at":"2022-06-11T17:32:54Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:17Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"9716f559ab7b333c80f9fb7209db58ce8a0fde42f3f553914472aa7b9ff74804","receipts":1}],"first_statement_date":"2022-06-11","related":[],"current":{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-c220b8-a2","payload":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-c220b8","subject_id":"gary-marcus","normalized_proposition":"Whenever AGI arrives, operations over variables, including storing, retrieving and comparing values, will be crucial to it.","kind":"conditional_forecast","conditions":["whenever AGI comes"],"original_horizon_text":null,"statement":{"excerpt":"👉operations over variables (including storing, retrieving and comparing values) will be crucial","locator":"prediction 3 of 4 (paragraph 57)","url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","date":"2022-06-11","document_title":"Does AI really need a paradigm shift?","observed_at":"2026-10-29T00:05:17Z"},"criteria":{"from_speaker":["whenever AGI comes","operations over variables","including storing, retrieving and comparing values","will be crucial"],"our_interpretation":["Treat arrival as a demonstrated system with broadly accepted human-level general cognitive capability. Broad acceptance is an evaluator safeguard.","Read operations over variables as reusable bindings between variables and values, with operations that store, retrieve, and compare those values in the system's reasoning.","The claim does not explicitly require a separate symbolic module. A neural implementation can qualify if it demonstrably performs the specified operations.","Incidental variables in implementation code do not establish a crucial cognitive mechanism.","Read crucial as central dependence. Test the arriving AGI without treating one implementation as proof about every possible AGI."],"support_condition":"The AGI condition is established, and architectural or mechanistic evidence shows that storing, retrieving, and comparing variable-bound values centrally enables its general capability.","contradiction_condition":"The AGI condition is established, and sufficiently complete mechanistic evidence shows that it attains general capability without the specified variable operations, or retains that capability when they are disabled."},"axis":"forecast_outcome","outcome":"unresolved","horizon_state":"open_horizon","evaluation_as_of":"2026-10-28","rationale":"On March 17, 2026, Google DeepMind announced a framework for measuring progress toward AGI and an initiative to develop evaluations [e1]. That announcement establishes neither a broadly accepted AGI demonstration nor whether variable operations centrally enable such a system's general capability, so it does not settle the conditional forecast [e1].","limitations":["The packet does not establish the AGI antecedent; this does not establish that AGI is absent.","No qualifying system has accessible mechanistic evidence establishing or excluding central dependence on the specified operations.","Ordinary memory use or success on variable-like tasks alone would not establish the required cognitive mechanism or its importance.","A candidate system-card PDF could not be inspected."],"evidence":[{"id":"e1","relation":"context","excerpt":"measure progress toward AGI","locator":"Article subtitle","url":"https://deepmind.google/blog/measuring-progress-toward-agi-a-cognitive-framework","publisher":"Google DeepMind","event_date":"2026-03-17","available_at":"2026-03-17","explanation":"This announces work on measuring progress. It does not demonstrate the antecedent or identify variable operations as necessary to an AGI."}],"what_would_resolve":"A dated, broadly accepted AGI demonstration, together with mechanistic evidence and interventions showing that reusable variable bindings and value storage, retrieval and comparison centrally enable its general capability—or that it achieves or retains that capability without those operations.","retrospective":true,"plain_reading":true},"payload_hash":"d7efe9fe42e7c309343587b2c148b9749a529d11cb1d6952c1dbf9b2f2716052","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T07:11:13Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-828976-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-does-ai-really-need-a-paradigm-shift-828976-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-828976-2x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T07:11:13Z","next_review_due_at":null,"horizon_now":"open_horizon"},"event_ids":["marcus-does-ai-really-need-a-paradigm-shift-c220b8-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-c94cb0","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-c94cb0-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-c94cb0#assessment-marcus-does-ai-really-need-a-paradigm-shift-c94cb0-a1"}],"claim":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-c94cb0","subject_id":"gary-marcus","revision":1,"content_hash":"89f64c9fa8abc6650e7a586df371fc59caa79118d4b32b9c583cd3278d8d5173","normalized_proposition":"Marcus puts a 90% probability on AI needing a paradigm shift to reach AGI.","kind":"forecast","target":"AI research toward AGI","conditions":[],"original_horizon_text":null,"topic_tags":["agi-timelines","scaling"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-does-ai-really-need-a-paradigm-shift-c94cb0-o1","statement_date":"2022-06-11","statement_date_precision":"day","excerpt":"Personally I’d put the probability that we need to shift at 90%, well above the 60% that Alexander suggests, and I would put the probability that we will need to embrace symbol-manipulation as part of the mix at 80%, more than double Alexander’s 34%.","locator":"paragraph 30","document":{"url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","title":"Does AI really need a paradigm shift?","publisher":"Marcus on AI","published_at":"2022-06-11T17:32:54Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:17Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"9716f559ab7b333c80f9fb7209db58ce8a0fde42f3f553914472aa7b9ff74804","receipts":1}],"first_statement_date":"2022-06-11","related":[],"current":{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-c94cb0-a1","payload":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-c94cb0","subject_id":"gary-marcus","normalized_proposition":"Marcus puts a 90% probability on AI needing a paradigm shift to reach AGI.","kind":"forecast","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Personally I’d put the probability that we need to shift at 90%, well above the 60% that Alexander suggests, and I would put the probability that we will need to embrace symbol-manipulation as part of the mix at 80%, more than double Alexander’s 34%.","locator":"paragraph 30","url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","date":"2022-06-11","document_title":"Does AI really need a paradigm shift?","observed_at":"2026-10-29T00:05:17Z"},"criteria":{"from_speaker":["90%"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"6888f47ab5fc45efff08ea8d8b4d47a275ab75444eb06aba210ab12fa75c9d03","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:10:54Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:10:54Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-does-ai-really-need-a-paradigm-shift-c94cb0-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-c94cb0-2","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-c94cb0-2-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-c94cb0-2#assessment-marcus-does-ai-really-need-a-paradigm-shift-c94cb0-2-a1"}],"claim":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-c94cb0-2","subject_id":"gary-marcus","revision":1,"content_hash":"93cba459192d07fd068d745a90e022f41038f39c124c7bd7d33abc690bf0c9ee","normalized_proposition":"Marcus puts an 80% probability on AGI needing symbol manipulation as part of the mix.","kind":"forecast","target":"AI research toward AGI","conditions":[],"original_horizon_text":null,"topic_tags":["agi-timelines"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-does-ai-really-need-a-paradigm-shift-c94cb0-2-o1","statement_date":"2022-06-11","statement_date_precision":"day","excerpt":"Personally I’d put the probability that we need to shift at 90%, well above the 60% that Alexander suggests, and I would put the probability that we will need to embrace symbol-manipulation as part of the mix at 80%, more than double Alexander’s 34%.","locator":"paragraph 30","document":{"url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","title":"Does AI really need a paradigm shift?","publisher":"Marcus on AI","published_at":"2022-06-11T17:32:54Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:17Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"9716f559ab7b333c80f9fb7209db58ce8a0fde42f3f553914472aa7b9ff74804","receipts":1}],"first_statement_date":"2022-06-11","related":[],"current":{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-c94cb0-2-a1","payload":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-c94cb0-2","subject_id":"gary-marcus","normalized_proposition":"Marcus puts an 80% probability on AGI needing symbol manipulation as part of the mix.","kind":"forecast","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Personally I’d put the probability that we need to shift at 90%, well above the 60% that Alexander suggests, and I would put the probability that we will need to embrace symbol-manipulation as part of the mix at 80%, more than double Alexander’s 34%.","locator":"paragraph 30","url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","date":"2022-06-11","document_title":"Does AI really need a paradigm shift?","observed_at":"2026-10-29T00:05:17Z"},"criteria":{"from_speaker":["80%"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"ee1526435a391faef6c0a158a0ebb5d4313022113bb9eb019b285500d7cb0772","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:10:54Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-does-ai-really-need-a-paradigm-shift-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:10:54Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-does-ai-really-need-a-paradigm-shift-c94cb0-2-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-fb3686","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-fb3686-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-does-ai-really-need-a-paradigm-shift-fb3686#assessment-marcus-does-ai-really-need-a-paradigm-shift-fb3686-a1"}],"claim":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-fb3686","subject_id":"gary-marcus","revision":1,"content_hash":"e6d0f9ca61fb59b09565acd77a8e1ba21c7cb4516456bbeeb2c66809c6713642","normalized_proposition":"Many of the harmful consequences of the webscraping/big-data approach to AI fall disproportionately on women and minorities.","kind":"observation","target":"webscraping/big data approach to AI","conditions":[],"original_horizon_text":null,"topic_tags":["safety"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-does-ai-really-need-a-paradigm-shift-fb3686-o1","statement_date":"2022-06-11","statement_date_precision":"day","excerpt":"so many of the unfortunate consequences of a webscraping/big data approach to AI are disproportionately borne by women and minorities.","locator":"paragraph 19","document":{"url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","title":"Does AI really need a paradigm shift?","publisher":"Marcus on AI","published_at":"2022-06-11T17:32:54Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:17Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"9716f559ab7b333c80f9fb7209db58ce8a0fde42f3f553914472aa7b9ff74804","receipts":1}],"first_statement_date":"2022-06-11","related":[],"current":{"assessment_id":"marcus-does-ai-really-need-a-paradigm-shift-fb3686-a1","payload":{"claim_id":"marcus-does-ai-really-need-a-paradigm-shift-fb3686","subject_id":"gary-marcus","normalized_proposition":"Many of the harmful consequences of the webscraping/big-data approach to AI fall disproportionately on women and minorities.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"so many of the unfortunate consequences of a webscraping/big data approach to AI are disproportionately borne by women and minorities.","locator":"paragraph 19","url":"https://garymarcus.substack.com/p/does-ai-really-need-a-paradigm-shift","date":"2022-06-11","document_title":"Does AI really need a paradigm shift?","observed_at":"2026-10-29T00:05:17Z"},"criteria":{"from_speaker":["so many of the unfortunate consequences","a webscraping/big data approach to AI","disproportionately borne by women and minorities"],"our_interpretation":["The plain reading is that multiple documented kinds of harm from this approach disproportionately affect women and minority groups. It does not require every harm to affect every such group.","Harm includes exclusion from training data, degrading representation, and discriminatory model outputs.","The observation concerns the research record and systems existing before the post. There is no standard reporting quarter for this subject; the target ends on the post date and has no stated beginning.","This assesses the distribution of documented harms, rather than whether web scraping alone caused them."],"support_condition":"Primary audits of existing datasets and models document multiple forms of disproportionate harm involving women and minority groups.","contradiction_condition":"Broad, comparable audits show that the alleged disparities are absent or reversed across the relevant harms, undermining the claimed recurring pattern."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The pre-post research documents several relevant disparities: exclusion of minority-associated language, sexualized representations of women, and degrading racial classifications. Together they support the ordinary qualitative reading of a recurring pattern. They do not establish that women and minorities bear most total AI harm, or that every harm falls disproportionately on those groups.","limitations":["Support depends on reading many qualitatively and including representational and data-exclusion harms.","The studies cover selected systems and probes, not a representative census of AI harms.","Some measured harms disproportionately affected men."],"evidence":[{"id":"e1","relation":"supports","excerpt":"disproportionately affected","locator":"Section 5.3, paragraph reporting dialect exclusion rates","url":"https://arxiv.org/html/2104.08758v2","publisher":"Dodge et al.; EMNLP 2021, arXiv version","event_date":"2021","available_at":"2021-09-30","explanation":"The audit reports removal rates of 42% for African American English and 32% for Hispanic-aligned English, against 6.2% for White-aligned English. This documents unequal exclusion from a web-scraped training corpus, using dialect as a proxy rather than identifying authors."},{"id":"e2","relation":"supports","excerpt":"returned pornographic imagery","locator":"Section 2.1, Misogyny and stereotypes","url":"https://arxiv.org/html/2110.01963v1","publisher":"Birhane, Prabhu and Kahembwe; arXiv","event_date":"2021-09-12","available_at":"2021-10-05","explanation":"The audit documents sexualized retrieval for ordinary feminine and demographic terms, including terms referring to minority women. This supports unequal representational harm, although the exploratory searches do not estimate its population-wide prevalence."},{"id":"e3","relation":"supports","excerpt":"significant disparities","locator":"Section 7, Broader Impacts, FairFace bias probes and Table 6","url":"https://arxiv.org/html/2103.00020v1","publisher":"Radford et al.; arXiv","event_date":"2021","available_at":"2021-02-26","explanation":"CLIP's audit reports nonhuman classifications for approximately 14% of Black images, versus under 8% for every other racial group. It also qualifies any universal gender claim: crime-related misclassification affected male images more often than female images, 16.5% versus 9.8%."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"15d69aed24457310145ffd63c28c99678fb72807cf900eca7bb0c11673e55179","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T06:59:19Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-fb3686-1x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-does-ai-really-need-a-paradigm-shift-fb3686-1x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-does-ai-really-need-a-paradigm-shift-fb3686-1x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e3","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T06:59:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-does-ai-really-need-a-paradigm-shift-fb3686-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-112099","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-112099-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-112099#assessment-marcus-what-does-it-mean-when-an-ai-fails-112099-a1"}],"claim":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-112099","subject_id":"gary-marcus","revision":1,"content_hash":"37757c4e1b83d768e63456538940649450a9fcc102223f3bdc09f12e6183435a","normalized_proposition":"Scott Alexander's finding that InstructGPT got four of six previously failed prompts right may have been a fluke.","kind":"observation","target":"InstructGPT","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-what-does-it-mean-when-an-ai-fails-112099-o1","statement_date":"2022-06-07","statement_date_precision":"day","excerpt":"Scott’s more marked success on six questions may have been a fluke.","locator":"paragraph 57","document":{"url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","publisher":"Marcus on AI","published_at":"2022-06-07T17:28:09Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:19Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"fa2a39c0800cc5475acd393215c920495373f319a6dd41cc57a65d2d3d491dbc","receipts":1}],"first_statement_date":"2022-06-07","related":[],"current":{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-112099-a1","payload":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-112099","subject_id":"gary-marcus","normalized_proposition":"Scott Alexander's finding that InstructGPT got four of six previously failed prompts right may have been a fluke.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Scott’s more marked success on six questions may have been a fluke.","locator":"paragraph 57","url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","date":"2022-06-07","document_title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","observed_at":"2026-10-29T00:05:19Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"84f5ac9be97027d48b5fb18ec4c7d005fcc4aef99a834d2122d151c3759b07bd","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:56:19Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-what-does-it-mean-when-an-ai-fails-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:56:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-what-does-it-mean-when-an-ai-fails-112099-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-739569","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-739569-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-739569#assessment-marcus-what-does-it-mean-when-an-ai-fails-739569-a1"}],"claim":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-739569","subject_id":"gary-marcus","revision":1,"content_hash":"fb17014e8aaccb4241c0c0765c0f48b40e09a72c0110fae3e93437bbcf22a833","normalized_proposition":"Because LLMs lack causal world models, they are not stable or trustworthy.","kind":"causal","target":"large language models","conditions":[],"original_horizon_text":null,"topic_tags":["reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-what-does-it-mean-when-an-ai-fails-739569-o1","statement_date":"2022-06-07","statement_date_precision":"day","excerpt":"The core problem that follows from that has always been that these systems aren’t stable or trustworthy; a crumbling house follows from a rotten foundation.","locator":"paragraph 50","document":{"url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","publisher":"Marcus on AI","published_at":"2022-06-07T17:28:09Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:19Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"fa2a39c0800cc5475acd393215c920495373f319a6dd41cc57a65d2d3d491dbc","receipts":1}],"first_statement_date":"2022-06-07","related":[],"current":{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-739569-a1","payload":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-739569","subject_id":"gary-marcus","normalized_proposition":"Because LLMs lack causal world models, they are not stable or trustworthy.","kind":"causal","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"The core problem that follows from that has always been that these systems aren’t stable or trustworthy; a crumbling house follows from a rotten foundation.","locator":"paragraph 50","url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","date":"2022-06-07","document_title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","observed_at":"2026-10-29T00:05:19Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"0719bea0f9077f022d83c158ece6581ed82b2c61369fd4e9f400fc47922a8f44","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:56:19Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-what-does-it-mean-when-an-ai-fails-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:56:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-what-does-it-mean-when-an-ai-fails-739569-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-7fce77","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-7fce77-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-7fce77#assessment-marcus-what-does-it-mean-when-an-ai-fails-7fce77-a1"}],"claim":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-7fce77","subject_id":"gary-marcus","revision":1,"content_hash":"2d6cf749369787329e4550c2cf4b9d13268ca0c7566b34bd1f4a3178aad9fa3d","normalized_proposition":"GPT-4 will very likely still have a fundamental lack of reliability, though it may take longer prompts to break.","kind":"forecast","target":"GPT-4","conditions":[],"original_horizon_text":null,"topic_tags":["reliability","scaling"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-what-does-it-mean-when-an-ai-fails-7fce77-o1","statement_date":"2022-06-07","statement_date_precision":"day","excerpt":"I would be very surprised if that fundamental lack of reliability did not also hold true for GPT-4. Maybe it will take three paragraphs in a prompt rather than one to break it, but a big Transformer alone does not a a theory of the world create.","locator":"paragraph 50","document":{"url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","publisher":"Marcus on AI","published_at":"2022-06-07T17:28:09Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:19Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"fa2a39c0800cc5475acd393215c920495373f319a6dd41cc57a65d2d3d491dbc","receipts":1}],"first_statement_date":"2022-06-07","related":[],"current":{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-7fce77-a1","payload":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-7fce77","subject_id":"gary-marcus","normalized_proposition":"GPT-4 will very likely still have a fundamental lack of reliability, though it may take longer prompts to break.","kind":"forecast","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"I would be very surprised if that fundamental lack of reliability did not also hold true for GPT-4. Maybe it will take three paragraphs in a prompt rather than one to break it, but a big Transformer alone does not a a theory of the world create.","locator":"paragraph 50","url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","date":"2022-06-07","document_title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","observed_at":"2026-10-29T00:05:19Z"},"criteria":{"from_speaker":["I would be very surprised","fundamental lack of reliability","also hold true for GPT-4","Maybe it will take three paragraphs in a prompt rather than one"],"our_interpretation":["The ordinary reading is that GPT-4 would retain recurring factual and reasoning failures serious enough to prevent unqualified reliance on its answers.","The statement allows improvement over earlier models and does not predict a particular error rate.","The three-paragraph remark is illustrative, not a firm prompt-length threshold.","The target is the forthcoming GPT-4 generation. The prediction can be resolved by evidence about that model after release; it does not require waiting for every future revision.","This assessment addresses reliability, not the separate philosophical assertion about whether Transformers create theories of the world."],"support_condition":"Dated evidence about released GPT-4 documents recurring hallucinations and reasoning errors that materially limit reliable use.","contradiction_condition":"Broad, reproducible testing establishes robust factual and reasoning reliability across the relevant tasks, including adversarial variations, such that the predicted recurring limitations are absent."},"axis":"forecast_outcome","outcome":"supported","horizon_state":"conclusive_event","evaluation_as_of":"2026-10-28","rationale":"OpenAI's launch-era technical report explicitly states that GPT-4 still hallucinates facts and makes reasoning errors. These are the recurring reliability problems the forecast anticipated. Substantial improvements and impressive capabilities do not contradict this forecast, which expressly allowed fewer failures. The evidence does not settle whether GPT-4 possesses a causal world model.","limitations":["Support applies to recurring factual and reasoning unreliability, not a claim that GPT-4 has no useful capabilities.","No numerical reliability threshold or causal explanation is established.","The decisive disclosure is from the developer."],"evidence":[{"id":"e1","relation":"supports","excerpt":"makes reasoning errors","locator":"Section 5, Limitations","url":"https://arxiv.org/html/2303.08774v1","publisher":"OpenAI / arXiv","event_date":"2023-03-15","available_at":"2023-03-15","explanation":"OpenAI's launch-era report explicitly acknowledges both predicted failure classes and advises safeguards where reliability matters. It also reports substantial factuality improvement, which is compatible with Marcus's allowance for fewer errors."},{"id":"e2","relation":"context","excerpt":"a true paradigm shift","locator":"Section 1, Introduction","url":"https://arxiv.org/html/2303.12712v1","publisher":"Microsoft Research researchers / arXiv","event_date":"2023-03-22","available_at":"2023-03-22","explanation":"The researchers interpret their experiments as substantial progress, cautioning against equating unreliability with an absence of capability. Their study concerns an early development model and also acknowledges limitations, so it does not contradict recurring failures in released GPT-4."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"07438d10d5bcbca0310685fdde2455f5562d8f35be0421508c52bd5f2ae490e7","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:47:24Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-204104-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-what-does-it-mean-when-an-ai-fails-204104-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-204104-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"company_disclosure","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:47:24Z","next_review_due_at":null,"horizon_now":"conclusive_event"},"event_ids":["marcus-what-does-it-mean-when-an-ai-fails-7fce77-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-9b99b3","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-9b99b3-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-9b99b3#assessment-marcus-what-does-it-mean-when-an-ai-fails-9b99b3-a1"}],"claim":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-9b99b3","subject_id":"gary-marcus","revision":2,"content_hash":"b6b584009c939bea3d0d67c8769777ce96415abc596faf406ee73595f173b6f2","normalized_proposition":"Marcus thinks it is scandalous that OpenAI never followed through on its promise of GPT-3 access to him and Davis and never replied to his request for DALL-E 2 access, and that Google Brain never replied to his request for Imagen access.","kind":"normative","target":"OpenAI and Google Brain","conditions":[{"text":"I think","type":"qualifier","from_source":true},{"text":"OpenAI's unkept GPT-3 access promise and unanswered DALL-E 2 request; Google Brain's unanswered Imagen request","type":"scope","from_source":true}],"original_horizon_text":null,"topic_tags":["other"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-what-does-it-mean-when-an-ai-fails-9b99b3-o1","statement_date":"2022-06-07","statement_date_precision":"day","excerpt":"I think all that’s scandalous.","locator":"paragraph 26","document":{"url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","publisher":"Marcus on AI","published_at":"2022-06-07T17:28:09Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:19Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"fa2a39c0800cc5475acd393215c920495373f319a6dd41cc57a65d2d3d491dbc","receipts":1}],"first_statement_date":"2022-06-07","related":[],"current":{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-9b99b3-a1","payload":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-9b99b3","subject_id":"gary-marcus","normalized_proposition":"Marcus thinks it is scandalous that OpenAI never followed through on its promise of GPT-3 access to him and Davis and never replied to his request for DALL-E 2 access, and that Google Brain never replied to his request for Imagen access.","kind":"normative","conditions":["I think","OpenAI's unkept GPT-3 access promise and unanswered DALL-E 2 request; Google Brain's unanswered Imagen request"],"original_horizon_text":null,"statement":{"excerpt":"I think all that’s scandalous.","locator":"paragraph 26","url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","date":"2022-06-07","document_title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","observed_at":"2026-10-29T00:05:19Z"},"criteria":{"from_speaker":["I think","OpenAI's unkept GPT-3 access promise and unanswered DALL-E 2 request; Google Brain's unanswered Imagen request"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"2518078a5b6bccae083d2bc9dcdaa1e4860a337adbf88813f439ba0fdc2cd004","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-what-does-it-mean-when-an-ai-fails-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-what-does-it-mean-when-an-ai-fails-9b99b3-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-a78bb4","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-a78bb4-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-a78bb4#assessment-marcus-what-does-it-mean-when-an-ai-fails-a78bb4-a1"}],"claim":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-a78bb4","subject_id":"gary-marcus","revision":1,"content_hash":"90f2f56ee019a1c73fb9e655ff625c108de814c1b92c6c9be7c3a0bd6c4b6b39","normalized_proposition":"LLMs on their own, without symbolic hybridization, are inherently weak at understanding the physical and psychological world.","kind":"observation","target":"large language models","conditions":[{"text":"unless hybridized with symbolic systems","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-what-does-it-mean-when-an-ai-fails-a78bb4-o1","statement_date":"2022-06-07","statement_date_precision":"day","excerpt":"Large language models are (unless hybridized with symbolic systems, a la Jurassic X) are, on their own, inherently weak at understanding the physical and psychological world.","locator":"paragraph 51","document":{"url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","publisher":"Marcus on AI","published_at":"2022-06-07T17:28:09Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:19Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"fa2a39c0800cc5475acd393215c920495373f319a6dd41cc57a65d2d3d491dbc","receipts":1}],"first_statement_date":"2022-06-07","related":[],"current":{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-a78bb4-a1","payload":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-a78bb4","subject_id":"gary-marcus","normalized_proposition":"LLMs on their own, without symbolic hybridization, are inherently weak at understanding the physical and psychological world.","kind":"observation","conditions":["unless hybridized with symbolic systems"],"original_horizon_text":null,"statement":{"excerpt":"Large language models are (unless hybridized with symbolic systems, a la Jurassic X) are, on their own, inherently weak at understanding the physical and psychological world.","locator":"paragraph 51","url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","date":"2022-06-07","document_title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","observed_at":"2026-10-29T00:05:19Z"},"criteria":{"from_speaker":["Large language models","unless hybridized with symbolic systems","on their own","inherently weak","understanding the physical and psychological world"],"our_interpretation":["The ordinary behavioral reading is a systematic weakness in reasoning about physical situations and people's beliefs, intentions and reactions, rather than a claim that models answer every such question incorrectly.","Inherently attributes the weakness to the unaided model approach; isolated mistakes are insufficient to establish that stronger component.","Symbolic solvers and external reasoning systems are excluded. Natural-language prompting alone does not count as symbolic hybridization.","The observation concerns models available by the post. Its contextual reference to continuing general phenomena supplies the August 2020-to-June 2022 historical interval.","Later models cannot directly decide this historical observation."],"support_condition":"Varied evaluations of contemporary unaided LLMs show systematic weaknesses in both physical and psychological reasoning, with evidence that those weaknesses belong to the approach rather than a particular benchmark or configuration.","contradiction_condition":"Contemporary unaided LLMs demonstrate robust, generalizable competence across varied physical and psychological reasoning tasks, including novel cases and controlled variations."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The evidence gives reasons for concern about physical reasoning and reasons to recognize substantial commonsense competence. Neither establishes the full claim about inherent weakness across both physical and psychological understanding in contemporary unaided LLMs. Calling it supported would generalize beyond the tested models and domains; calling it contradicted would treat benchmark success as broad robustness.","limitations":["TRIP tests a narrower model population than the frontier generative systems discussed in the post.","PaLM's favorable results do not by themselves demonstrate robust understanding.","The word inherently makes the claim stronger than an observation of particular errors."],"evidence":[{"id":"e1","relation":"context","excerpt":"supporting evidence.","locator":"Abstract; experimental population checked in full-paper Section 4","url":"https://arxiv.org/abs/2109.04947","publisher":"Shane Storks, Qiaozi Gao, Yichi Zhang and Joyce Chai / arXiv","event_date":"2021","available_at":"2022-05-10","explanation":"TRIP documents difficulties connecting physical story judgments to supporting states and conflicts. Its BERT-family systems and physical-only coverage do not establish the complete claim about contemporary leading LLMs and both domains."},{"id":"e2","relation":"context","excerpt":"reasoning tasks.","locator":"Conclusion; reasoning methods in Section 6.3","url":"https://arxiv.org/html/2204.02311v3","publisher":"Google researchers / arXiv","event_date":"2022","available_at":"2022-04-19","explanation":"PaLM's commonsense results show meaningful competence using natural-language prompting, limiting a blanket reading of weakness. The combined statement also covers calculator-assisted arithmetic, and does not establish robust competence across both physical and psychological reasoning."}],"what_would_resolve":"Broad, controlled evaluations of the contemporaneous models across both domains, with novel task variations and documented absence of symbolic assistance, together with evidence distinguishing systematic model limitations from benchmark-specific failures.","retrospective":true,"plain_reading":true},"payload_hash":"19df496e443b745a55dfb43313c55a44dedc42dd8b6eb4193e70e9d0926891ad","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:47:24Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-204104-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-what-does-it-mean-when-an-ai-fails-204104-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-204104-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:47:24Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-what-does-it-mean-when-an-ai-fails-a78bb4-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-ae0bd9","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-ae0bd9-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-ae0bd9#assessment-marcus-what-does-it-mean-when-an-ai-fails-ae0bd9-a1"}],"claim":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-ae0bd9","subject_id":"gary-marcus","revision":2,"content_hash":"08ae0365cccecb5fe9e3c7186d12e21e4d1efe7763778cef27c0a282399a1e25","normalized_proposition":"Marcus sees no reason whatsoever to think that the underlying problem of these models, a lack of cognitive models of the world, has been remedied.","kind":"observation","target":"large language models","conditions":[{"text":"I see no reason whatsoever to think","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-what-does-it-mean-when-an-ai-fails-ae0bd9-o1","statement_date":"2022-06-07","statement_date_precision":"day","excerpt":"But I see no reason whatsoever to think that the underlying problem — a lack of cognitive models of the world —have been remedied.","locator":"paragraph 38","document":{"url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","publisher":"Marcus on AI","published_at":"2022-06-07T17:28:09Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:19Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"fa2a39c0800cc5475acd393215c920495373f319a6dd41cc57a65d2d3d491dbc","receipts":1}],"first_statement_date":"2022-06-07","related":[],"current":{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-ae0bd9-a1","payload":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-ae0bd9","subject_id":"gary-marcus","normalized_proposition":"Marcus sees no reason whatsoever to think that the underlying problem of these models, a lack of cognitive models of the world, has been remedied.","kind":"observation","conditions":["I see no reason whatsoever to think"],"original_horizon_text":null,"statement":{"excerpt":"But I see no reason whatsoever to think that the underlying problem — a lack of cognitive models of the world —have been remedied.","locator":"paragraph 38","url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","date":"2022-06-07","document_title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","observed_at":"2026-10-29T00:05:19Z"},"criteria":{"from_speaker":["I see no reason whatsoever to think"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"b0c165e4c5de4d3a4b1f547fe9c1c3cb278756afeb07b3a3d680396f1c1b1283","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-what-does-it-mean-when-an-ai-fails-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-what-does-it-mean-when-an-ai-fails-ae0bd9-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-b49d13","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-b49d13-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-b49d13#assessment-marcus-what-does-it-mean-when-an-ai-fails-b49d13-a1"}],"claim":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-b49d13","subject_id":"gary-marcus","revision":1,"content_hash":"995cc2aca48e697a4206dc61226bc61a0f418b60eabfc9cf148caa7a23dfde6c","normalized_proposition":"Current LLMs have no real physical, social, or psychological understanding; they are richer parroting of larger corpora.","kind":"observation","target":"large language models","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-what-does-it-mean-when-an-ai-fails-b49d13-o1","statement_date":"2022-06-07","statement_date_precision":"day","excerpt":"Either way, though, there is still no real physical, social, or psychological understanding, just richer parroting of larger corpora (and some reinforcement learning to keep the system more on check).","locator":"paragraph 47","document":{"url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","publisher":"Marcus on AI","published_at":"2022-06-07T17:28:09Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:19Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"fa2a39c0800cc5475acd393215c920495373f319a6dd41cc57a65d2d3d491dbc","receipts":1}],"first_statement_date":"2022-06-07","related":[],"current":{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-b49d13-a1","payload":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-b49d13","subject_id":"gary-marcus","normalized_proposition":"Current LLMs have no real physical, social, or psychological understanding; they are richer parroting of larger corpora.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Either way, though, there is still no real physical, social, or psychological understanding, just richer parroting of larger corpora (and some reinforcement learning to keep the system more on check).","locator":"paragraph 47","url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","date":"2022-06-07","document_title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","observed_at":"2026-10-29T00:05:19Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"4391e07f5e01eb6965bf0d8c247ea8781c5e8d72f107ede8cb43bfc875e90b65","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:56:19Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-what-does-it-mean-when-an-ai-fails-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:56:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-what-does-it-mean-when-an-ai-fails-b49d13-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-c64e20","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-c64e20-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-c64e20#assessment-marcus-what-does-it-mean-when-an-ai-fails-c64e20-a1"}],"claim":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-c64e20","subject_id":"gary-marcus","revision":2,"content_hash":"3fe950db3ffe5baa96c5bbab8a01f626c43297d9e12eae4c1226f10d432eecb3","normalized_proposition":"If GPT-4 is basically the same architecture as GPT-3 but with more data, it will not be proficient enough to have developed a genuine causal model of the world, though Marcus and Davis may find fewer absolute howlers per attempt than with GPT-3.","kind":"conditional_forecast","target":"GPT-4","conditions":[{"text":"assuming GPT-4 is basically the same architecture but with more data","type":"antecedent","from_source":true},{"text":"maybe fewer absolute howlers per attempt","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities","scaling"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-what-does-it-mean-when-an-ai-fails-c64e20-o1","statement_date":"2022-06-07","statement_date_precision":"day","excerpt":"Maybe we will find fewer absolute howlers per attempt than we did with GPT-3, but it still won’t be so proficient as to have developed a genuine causal model of the world.","locator":"paragraph 49","document":{"url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","publisher":"Marcus on AI","published_at":"2022-06-07T17:28:09Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:19Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"fa2a39c0800cc5475acd393215c920495373f319a6dd41cc57a65d2d3d491dbc","receipts":1}],"first_statement_date":"2022-06-07","related":[],"current":{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-c64e20-a1","payload":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-c64e20","subject_id":"gary-marcus","normalized_proposition":"If GPT-4 is basically the same architecture as GPT-3 but with more data, it will not be proficient enough to have developed a genuine causal model of the world, though Marcus and Davis may find fewer absolute howlers per attempt than with GPT-3.","kind":"conditional_forecast","conditions":["assuming GPT-4 is basically the same architecture but with more data","maybe fewer absolute howlers per attempt"],"original_horizon_text":null,"statement":{"excerpt":"Maybe we will find fewer absolute howlers per attempt than we did with GPT-3, but it still won’t be so proficient as to have developed a genuine causal model of the world.","locator":"paragraph 49","url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","date":"2022-06-07","document_title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","observed_at":"2026-10-29T00:05:19Z"},"criteria":{"from_speaker":["assuming GPT-4 is basically the same architecture but with more data","maybe fewer absolute howlers per attempt"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"c68c66b0f61dc647fd31e17ab5efe00165bd7bc2a896426107c1ae5eb2abae7f","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-what-does-it-mean-when-an-ai-fails-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-what-does-it-mean-when-an-ai-fails-c64e20-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-cdba9a","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-cdba9a-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-cdba9a#assessment-marcus-what-does-it-mean-when-an-ai-fails-cdba9a-a1"}],"claim":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-cdba9a","subject_id":"gary-marcus","revision":2,"content_hash":"fa0fa6313322f49096628f8c70ecf45bd6a00902a5a1f654504951079621bb0d","normalized_proposition":"As Davis has found, across the full set of their Technology Review questions, the difference between GPT-3's earlier version and InstructGPT is far more modest than one might have expected: fewer outright wacky answers and some old answers now correct, but some backsliding too.","kind":"observation","target":"InstructGPT","conditions":[{"text":"across the full set of our Technology Review questions","type":"scope","from_source":true},{"text":"far more modest than one might have expected","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities","reliability"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-what-does-it-mean-when-an-ai-fails-cdba9a-o1","statement_date":"2022-06-07","statement_date_precision":"day","excerpt":"Interestingly, as Ernie has just discovered, across the full set of our Technology Review questions the difference from GPT-3’s earlier version to the current version (InstructGPT) is far more modest than one might have expected: fewer outright wacky answers, some old answers now correct, but some backsliding, too.","locator":"paragraph 57","document":{"url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","publisher":"Marcus on AI","published_at":"2022-06-07T17:28:09Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:19Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"fa2a39c0800cc5475acd393215c920495373f319a6dd41cc57a65d2d3d491dbc","receipts":1}],"first_statement_date":"2022-06-07","related":[],"current":{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-cdba9a-a1","payload":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-cdba9a","subject_id":"gary-marcus","normalized_proposition":"As Davis has found, across the full set of their Technology Review questions, the difference between GPT-3's earlier version and InstructGPT is far more modest than one might have expected: fewer outright wacky answers and some old answers now correct, but some backsliding too.","kind":"observation","conditions":["across the full set of our Technology Review questions","far more modest than one might have expected"],"original_horizon_text":null,"statement":{"excerpt":"Interestingly, as Ernie has just discovered, across the full set of our Technology Review questions the difference from GPT-3’s earlier version to the current version (InstructGPT) is far more modest than one might have expected: fewer outright wacky answers, some old answers now correct, but some backsliding, too.","locator":"paragraph 57","url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","date":"2022-06-07","document_title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","observed_at":"2026-10-29T00:05:19Z"},"criteria":{"from_speaker":["across the full set of our Technology Review questions","GPT-3’s earlier version to the current version (InstructGPT)","far more modest than one might have expected","fewer outright wacky answers","some old answers now correct","some backsliding, too"],"our_interpretation":["The plain reading is a limited overall improvement on the complete question set, with fewer plainly nonsensical responses, some corrected failures, and some newly incorrect answers.","The surrounding discussion supplies Alexander’s six-question comparison as the reference for expectations. It does not supply a numerical threshold for modest improvement.","The target is the comparison newly reported around the post date. I use June 1–7, 2022 for this early-June observation; the earlier GPT-3 answers serve as its historical baseline.","Backsliding means an earlier correct response becoming incorrect on the same question. It does not by itself establish a decline in the model’s average capability."],"support_condition":"An admissible comparison covering the complete question set shows a limited aggregate improvement, fewer plainly nonsensical answers, at least one corrected earlier failure, and at least one earlier success becoming incorrect.","contradiction_condition":"An admissible complete-set comparison shows a clearly large aggregate improvement under consistent scoring, or establishes that one of the claimed directional changes did not occur."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The search located the directly relevant full-set report, but Marcus coauthored it. The supplied rules prohibit using his own writing to establish that he was right. Alexander’s independent test reports four corrected answers among six earlier failures, but that selection cannot reveal backsliding or establish the overall change across the complete set. The admissible evidence therefore does not settle the claim. This is an evidence limitation, not evidence that Marcus was wrong.","limitations":["The source restriction excludes the report most directly associated with the claim.","A small set selected from earlier failures cannot determine complete-set performance.","Modest improvement and wacky answers lack numerical definitions."],"evidence":[{"id":"e1","relation":"context","excerpt":"Of these six prompts that GPT-3 original failed, GPT-3 advanced gets four unambiguously right.","locator":"Section Marcus vs. GPT, Round 2, paragraph summarizing the six retested prompts","url":"https://www.astralcodexten.com/p/my-bet-ai-size-solves-flubs","publisher":"Scott Alexander, Astral Codex Ten","event_date":"2022-06","available_at":"2022-06-07","explanation":"Alexander reports corrections on a subset of earlier failures. This cannot establish the complete-set improvement or reveal regressions on earlier successes, so it neither settles nor contradicts Marcus’s full claim."}],"what_would_resolve":"An independent audit of authenticated outputs from the original and early-June 2022 tests, covering the complete question set and applying consistent scoring, with matched counts of corrections, regressions, unchanged failures, and nonsensical responses.","retrospective":true,"plain_reading":true},"payload_hash":"ab9a71cd06798f2cbe99b054a6ab2e7175aeb1a097da0cf7b7d431823399ebb8","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:47:24Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-204104-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-what-does-it-mean-when-an-ai-fails-204104-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-204104-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"independent_measurement","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:47:24Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-what-does-it-mean-when-an-ai-fails-cdba9a-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-d02c78","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-d02c78-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-d02c78#assessment-marcus-what-does-it-mean-when-an-ai-fails-d02c78-a1"}],"claim":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-d02c78","subject_id":"gary-marcus","revision":2,"content_hash":"8f45544e6875f237e7a66ed32a4eeaeed71c751cbca15bb45804bc3d8e0c38c2","normalized_proposition":"So far as Marcus can tell, all the core claims he and Davis made in Technology Review still hold true nearly two years and many even larger models later; some of their specific examples have been remedied, but the general problems have not.","kind":"observation","target":"large language models","conditions":[{"text":"so far as I can tell","type":"qualifier","from_source":true},{"text":"nearly two years and many even larger models later","type":"scope","from_source":true}],"original_horizon_text":null,"topic_tags":["reliability","capabilities","scaling"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-what-does-it-mean-when-an-ai-fails-d02c78-o1","statement_date":"2022-06-07","statement_date_precision":"day","excerpt":"The core claims that Ernie and I made in Technology Review still all, so far as I can tell, hold true, nearly two years and many even larger models later. Yes, some of the specific examples we pointed out have been remedied, but the general problems have not.","locator":"paragraph 29","document":{"url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","publisher":"Marcus on AI","published_at":"2022-06-07T17:28:09Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:19Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"fa2a39c0800cc5475acd393215c920495373f319a6dd41cc57a65d2d3d491dbc","receipts":1}],"first_statement_date":"2022-06-07","related":[],"current":{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-d02c78-a1","payload":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-d02c78","subject_id":"gary-marcus","normalized_proposition":"So far as Marcus can tell, all the core claims he and Davis made in Technology Review still hold true nearly two years and many even larger models later; some of their specific examples have been remedied, but the general problems have not.","kind":"observation","conditions":["so far as I can tell","nearly two years and many even larger models later"],"original_horizon_text":null,"statement":{"excerpt":"The core claims that Ernie and I made in Technology Review still all, so far as I can tell, hold true, nearly two years and many even larger models later. Yes, some of the specific examples we pointed out have been remedied, but the general problems have not.","locator":"paragraph 29","url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","date":"2022-06-07","document_title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","observed_at":"2026-10-29T00:05:19Z"},"criteria":{"from_speaker":["core claims","still all","so far as I can tell","nearly two years","some of the specific examples ... have been remedied","the general problems have not"],"our_interpretation":["The context identifies six general problems: biological, physical, social and practical reasoning, object and individual tracking, and non sequiturs.","The ordinary reading is that these remained substantive weaknesses, despite improvements on particular examples. It does not require the models to fail every test.","Because this describes the current model generation without a reporting period, I use 2022 through the post date as the latest public evaluation period.","The qualification about Marcus's knowledge does not remove the factual assertion. However, evidence about only some categories cannot establish his assertion about all six."],"support_condition":"Public evaluations of the relevant model generation establish continuing substantive failures across all six categories, while allowing improvements on individual examples.","contradiction_condition":"Robust evaluations establish that at least one of the six general problems had been resolved across varied tests by the post date. Better average scores or corrected examples alone do not suffice."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The records support continuing mistakes and also show substantial progress. They do not establish Marcus's stronger assertion that all six general problems remained unresolved. Nor do the improved benchmark results establish that any whole category had been solved. The breadth of the claim remains unverified.","limitations":["This is a gap in the retrieved evidence, not evidence that the categories had been solved.","Benchmark averages do not directly measure the six categories as Marcus defined them.","The dated disclosures identify public capability snapshots, not the exact dates each experiment ran."],"evidence":[{"id":"e1","relation":"supports","excerpt":"InstructGPT still makes simple mistakes.","locator":"Introduction, summary of limitations; section 4, qualitative results","url":"https://arxiv.org/html/2203.02155v1","publisher":"OpenAI researchers / arXiv","event_date":"2022-03-04","available_at":"2022-03-04","explanation":"The reported model still accepted false premises, invented facts and struggled with multiple constraints. This supports persistent reliability problems, but does not establish failures across all six categories."},{"id":"e2","relation":"context","excerpt":"average performance","locator":"Language Understanding and Generation, BIG-bench comparison","url":"https://research.google/blog/pathways-language-model-palm-scaling-to-540-billion-parameters-for-breakthrough-performance","publisher":"Google Research","event_date":"2022-04-04","available_at":"2022-04-04","explanation":"Google reported strong average performance on a subset of 58 BIG-bench tasks. This is material evidence of progress, but the aggregate does not establish that any entire Marcus category had been resolved."}],"what_would_resolve":"Independent tests of the models available by June 7, 2022, covering all six categories with varied examples, or robust evidence that one complete category had been resolved.","retrospective":true,"plain_reading":true},"payload_hash":"7daa73a907a99f3714a401fc9a88a175c00313d753d0b5fac985b1e86c9ed799","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:47:03Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-d02c78-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-what-does-it-mean-when-an-ai-fails-d02c78-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-d02c78-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:47:03Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-what-does-it-mean-when-an-ai-fails-d02c78-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-d5182b","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-d5182b-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-d5182b#assessment-marcus-what-does-it-mean-when-an-ai-fails-d5182b-a2"}],"claim":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-d5182b","subject_id":"gary-marcus","revision":1,"content_hash":"b9dfdced78e090cdd40a5e8e7de0ef73a053f60e58e3ef6dee26ad333124f65c","normalized_proposition":"Newer LLMs still lack genuine comprehension and are therefore still very easy for anyone to break.","kind":"causal","target":"large language models","conditions":[],"original_horizon_text":null,"topic_tags":["reliability","capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-what-does-it-mean-when-an-ai-fails-d5182b-o1","statement_date":"2022-06-07","statement_date_precision":"day","excerpt":"But they still don’t convey genuine comprehension, and so they are still very easy for Ernie and me (or anyone else who cares to try) to break.","locator":"paragraph 38","document":{"url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","publisher":"Marcus on AI","published_at":"2022-06-07T17:28:09Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:19Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"fa2a39c0800cc5475acd393215c920495373f319a6dd41cc57a65d2d3d491dbc","receipts":1}],"first_statement_date":"2022-06-07","related":[],"current":{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-d5182b-a2","payload":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-d5182b","subject_id":"gary-marcus","normalized_proposition":"Newer LLMs still lack genuine comprehension and are therefore still very easy for anyone to break.","kind":"causal","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"But they still don’t convey genuine comprehension, and so they are still very easy for Ernie and me (or anyone else who cares to try) to break.","locator":"paragraph 38","url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","date":"2022-06-07","document_title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","observed_at":"2026-10-29T00:05:19Z"},"criteria":{"from_speaker":["still don’t convey genuine comprehension","and so","very easy","anyone else who cares to try","to break"],"our_interpretation":["The ordinary behavioral reading is that interested users could readily elicit clear factual or commonsense errors from the contemporary models.","The words 'and so' additionally attribute those failures to missing comprehension. Demonstrating errors alone does not establish that explanation.","The historical scope is the continuation of the problems identified on August 22, 2020 through the post date, including the successors discussed in the post.","No numerical failure rate, testing budget, or independent measure of comprehension is supplied."],"support_condition":"Independent tests show readily reproducible failures in contemporary models, and evidence distinguishes absent comprehension as their cause from alternative explanations.","contradiction_condition":"Representative tests show robust performance against ordinary attempts to elicit these failures, or controlled evidence undermines the proposed explanation that missing comprehension causes them."},"axis":"causal_assessment","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"TruthfulQA's 2021 results document factual errors in GPT-3-era models; the best tested model was truthful on 58% of its misconception-focused questions [e1]. The report published March 4, 2022 documents InstructGPT fabricating facts, failing instructions and accepting false premises, alongside improved instruction following [e2]. This supports continued fallibility, but only partially addresses the behavioral claim: neither report measures how readily ordinary users could find failures across the relevant models [e1, e2]. Neither independently measures comprehension or establishes its absence as the cause of those failures, leaving the causal claim unresolved [e1, e2].","limitations":["Benchmark scores and qualitative examples do not establish that anyone could easily elicit failures across all relevant models [e1, e2].","The evidence does not distinguish missing comprehension from competing explanations for the errors [e1, e2]."],"evidence":[{"id":"e1","relation":"supports","excerpt":"truthful on 58% of questions","locator":"Abstract","url":"https://arxiv.org/abs/2109.07958","publisher":"Stephanie Lin, Jacob Hilton and Owain Evans / arXiv","event_date":"2021","available_at":"2022-05-08","explanation":"The benchmark documents substantial factual fallibility in the tested GPT-3-era models. It supports the behavioral component, but measures neither users' effort to discover failures nor their proposed cognitive cause."},{"id":"e2","relation":"supports","excerpt":"InstructGPT still makes simple mistakes.","locator":"Section 4.3, Qualitative results","url":"https://arxiv.org/html/2203.02155v1","publisher":"OpenAI researchers / arXiv","event_date":"2022","available_at":"2022-03-04","explanation":"The report describes false-premise acceptance and failures under multiple constraints, extending the behavioral evidence to InstructGPT. It also reports improved instruction following and generalization, without establishing whether comprehension is absent."}],"what_would_resolve":"Reproducible tests of the contemporary model versions measuring ordinary users' attempts and time needed to find failures, plus controlled studies using an independent measure of comprehension to test whether its absence explains those failures rather than competing causes.","retrospective":true,"plain_reading":true},"payload_hash":"3d9dd00d6dd3781f3ecccb16d59c73add9bb3d599a0bf630a1b62e0d0402191a","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:56:44Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-d02c78-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-what-does-it-mean-when-an-ai-fails-d02c78-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-204104-3x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:56:44Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-what-does-it-mean-when-an-ai-fails-d5182b-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-e02fad","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-e02fad-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-e02fad#assessment-marcus-what-does-it-mean-when-an-ai-fails-e02fad-a2"}],"claim":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-e02fad","subject_id":"gary-marcus","revision":1,"content_hash":"fbbe18141921fe74fd21f39088eef3339ffba2eb4cbbd0dd76b137f1232daad8","normalized_proposition":"If given unrestricted access to GPT-4 and it is basically the same architecture with more data, Marcus and Davis will find many failures in physical, temporal, and causal reasoning within a day.","kind":"conditional_forecast","target":"GPT-4","conditions":[{"text":"OpenAI gives unrestricted access to GPT-4","type":"antecedent","from_source":true},{"text":"GPT-4 is basically the same architecture with more data","type":"antecedent","from_source":true}],"original_horizon_text":"within a day of playing around with it","topic_tags":["reliability","capabilities","scaling"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-what-does-it-mean-when-an-ai-fails-e02fad-o1","statement_date":"2022-06-07","statement_date_precision":"day","excerpt":"I am willing to bet SlateStarCodex now (terms to be negotiated) that if OpenAI gives us unrestricted access to GPT-4, whenever that is released, and assuming that is basically the same architecture but with more data, that within a day of playing around with it, Ernie and I will still be able find lots of examples of failures in physical reasoning, temporal reasoning, causal reasoning, and so forth.","locator":"paragraph 48","document":{"url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","publisher":"Marcus on AI","published_at":"2022-06-07T17:28:09Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:19Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"fa2a39c0800cc5475acd393215c920495373f319a6dd41cc57a65d2d3d491dbc","receipts":1}],"first_statement_date":"2022-06-07","related":[],"current":{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-e02fad-a2","payload":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-e02fad","subject_id":"gary-marcus","normalized_proposition":"If given unrestricted access to GPT-4 and it is basically the same architecture with more data, Marcus and Davis will find many failures in physical, temporal, and causal reasoning within a day.","kind":"conditional_forecast","conditions":["OpenAI gives unrestricted access to GPT-4","GPT-4 is basically the same architecture with more data"],"original_horizon_text":"within a day of playing around with it","statement":{"excerpt":"I am willing to bet SlateStarCodex now (terms to be negotiated) that if OpenAI gives us unrestricted access to GPT-4, whenever that is released, and assuming that is basically the same architecture but with more data, that within a day of playing around with it, Ernie and I will still be able find lots of examples of failures in physical reasoning, temporal reasoning, causal reasoning, and so forth.","locator":"paragraph 48","url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","date":"2022-06-07","document_title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","observed_at":"2026-10-29T00:05:19Z"},"criteria":{"from_speaker":["if OpenAI gives us unrestricted access to GPT-4","whenever that is released","assuming that is basically the same architecture but with more data","within a day of playing around with it","Ernie and I","lots of examples","physical reasoning, temporal reasoning, causal reasoning"],"our_interpretation":["Both access and architecture conditions must be established before scoring the forecast.","The named testers are Marcus and Ernest Davis. Other researchers' results cannot substitute for their first day of testing.","The deadline is relative to qualifying access and the start of testing; the post supplies no calendar deadline.","Lots means numerous distinct failures across the stated kinds of reasoning, but no exact count was negotiated.","Unrestricted access is not assumed to mean ordinary access to a rate-limited public service."],"support_condition":"Evidence establishes both antecedents and a timestamped first-day testing record shows Marcus and Davis finding numerous distinct failures across the specified reasoning areas.","contradiction_condition":"Evidence establishes both antecedents and a sufficiently complete record of their first day shows that they did not find numerous qualifying failures."},"axis":"forecast_outcome","outcome":"unresolved","horizon_state":"open_horizon","evaluation_as_of":"2026-10-28","rationale":"OpenAI's March 14, 2023 launch disclosure describes capped ChatGPT Plus access and a staged API rollout with default rate limits [e1]. It does not establish unrestricted access for Marcus and Davis, verify the same-architecture-with-more-data condition, or document their first day of testing [e1]. The supplied evidence therefore establishes neither the conditions needed to start the one-day window nor a qualifying testing result; the forecast remains unresolved with an open horizon [e1].","limitations":["The public rollout disclosure does not establish whether Marcus and Davis separately received qualifying access [e1].","The supplied record does not verify the architecture and additional-data condition or provide timestamped first-day testing logs.","The proposed bet leaves 'lots,' 'unrestricted access' and the precise meaning of a day undefined."],"evidence":[{"id":"e1","relation":"context","excerpt":"start inviting some developers today","locator":"API section","url":"https://openai.com/index/gpt-4-research","publisher":"OpenAI","event_date":"2023-03-14","available_at":"2023-03-14","explanation":"The launch describes a staged public rollout. It supplies no confirmation that Marcus and Davis received access satisfying the forecast's unrestricted-access condition."}],"what_would_resolve":"Documentation establishing both antecedents, followed by independently checkable logs of Marcus and Davis's first qualifying day, including start time, model version, prompts, outputs and distinct failures across the specified reasoning areas. Applying the result also requires clear interpretations of 'lots,' 'unrestricted access' and the one-day window.","retrospective":true,"plain_reading":true},"payload_hash":"44ffa27bb07be956873ddaeafe6815a7bb598bf747baf715585a7c358097533e","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:56:44Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-d02c78-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-what-does-it-mean-when-an-ai-fails-d02c78-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-204104-3x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:56:44Z","next_review_due_at":null,"horizon_now":"open_horizon"},"event_ids":["marcus-what-does-it-mean-when-an-ai-fails-e02fad-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-e6235f","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-e6235f-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-e6235f#assessment-marcus-what-does-it-mean-when-an-ai-fails-e6235f-a1"}],"claim":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-e6235f","subject_id":"gary-marcus","revision":2,"content_hash":"7790982245bf1d6e94b257e4f79fd67a996fdc84d25a5cd4ccc0d519cc063251","normalized_proposition":"The newer models' improvements, such as they are, come primarily from their larger and larger sets of data about how humans use word sequences, which are certainly helpful for pattern-matching machines.","kind":"causal","target":"large language models","conditions":[{"text":"such as they are","type":"qualifier","from_source":true},{"text":"primarily","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["scaling","capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-what-does-it-mean-when-an-ai-fails-e6235f-o1","statement_date":"2022-06-07","statement_date_precision":"day","excerpt":"The improvements, such as they are, come, primarily because the newer models have larger and larger sets of data about how human beings use word sequences, and bigger word sequences are certainly helpful for pattern matching machines.","locator":"paragraph 38","document":{"url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","publisher":"Marcus on AI","published_at":"2022-06-07T17:28:09Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:19Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"fa2a39c0800cc5475acd393215c920495373f319a6dd41cc57a65d2d3d491dbc","receipts":1}],"first_statement_date":"2022-06-07","related":[],"current":{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-e6235f-a1","payload":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-e6235f","subject_id":"gary-marcus","normalized_proposition":"The newer models' improvements, such as they are, come primarily from their larger and larger sets of data about how humans use word sequences, which are certainly helpful for pattern-matching machines.","kind":"causal","conditions":["such as they are","primarily"],"original_horizon_text":null,"statement":{"excerpt":"The improvements, such as they are, come, primarily because the newer models have larger and larger sets of data about how human beings use word sequences, and bigger word sequences are certainly helpful for pattern matching machines.","locator":"paragraph 38","url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","date":"2022-06-07","document_title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","observed_at":"2026-10-29T00:05:19Z"},"criteria":{"from_speaker":["The improvements, such as they are","come, primarily because","larger and larger sets of data","how human beings use word sequences"],"our_interpretation":["The claim concerns the main cause of improvements from GPT-2 through GPT-3 and the successors discussed by June 2022.","Primarily means that increasing the quantity of training text was the leading explanation, rather than merely one useful contributor.","The ordinary reading concerns training-data quantity. It does not automatically include model parameter count, different training objectives, human feedback or prompting changes.","The sentence about pattern matching is not treated as a separately established theory of the models' internal representations."],"support_condition":"Controlled comparisons and a cross-generation accounting show that larger text datasets explain the dominant share of the relevant gains after accounting for model size, compute, data quality, prompting and training methods.","contradiction_condition":"Controlled comparisons and a cross-generation accounting establish that another factor, or factors other than data quantity together, predominantly explain those gains."},"axis":"causal_assessment","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"More training data clearly contributed to progress. The scaling studies also identify parameter count and compute as important, while InstructGPT demonstrates gains from changing the training procedure. These findings pull in different directions for particular improvements. They do not establish which factor primarily explains the whole progression Marcus describes, so they do not settle his causal claim.","limitations":["An optimal scaling prescription is not a decomposition of historical progress.","The Chinchilla comparison changes more than data quantity.","Instruction-following preference is not interchangeable with factual accuracy.","Evidence for and against individual components does not settle the dominant-cause assertion."],"evidence":[{"id":"e9","relation":"supports","excerpt":"equal proportions.","locator":"Section 1, Introduction; sections 3 and 4","url":"https://arxiv.org/html/2203.15556v1","publisher":"DeepMind researchers / arXiv","event_date":"2022-03-29","available_at":"2022-03-29","explanation":"The experiments support a substantial role for more training data, including a smaller model trained on more tokens outperforming larger predecessors. They do not establish that data quantity predominantly caused the entire set of gains Marcus discusses; the paper also reports changes in the final model's training setup."},{"id":"e10","relation":"context","excerpt":"capacity-limited.","locator":"Section 4.1; section 1.1 summary of scaling results","url":"https://arxiv.org/html/2001.08361v1","publisher":"OpenAI researchers / arXiv","event_date":"2020-01-23","available_at":"2020-01-23","explanation":"The study identifies model size, dataset size and compute as distinct contributors to language-model loss. It cautions against a data-only explanation, but does not apportion the downstream gains of the full model sequence in Marcus's claim."},{"id":"e11","relation":"contradicts","excerpt":"better at following instructions","locator":"Opening explanation of reinforcement learning from human feedback; Methods","url":"https://openai.com/index/instruction-following","publisher":"OpenAI","event_date":"2022-01-27","available_at":"2022-01-27","explanation":"OpenAI attributes these gains to supervised demonstrations and preference-based reinforcement learning applied to GPT-3, using little additional data relative to pretraining. This challenges the proposed explanation for this particular successor, but cannot determine the primary cause of all the improvements under discussion."}],"what_would_resolve":"Matched ablations and a clearly scoped comparison of the relevant model generations that quantify the contributions of data quantity, model size, compute and training methods to the same performance measures.","retrospective":true,"plain_reading":true},"payload_hash":"979522e47f6216084175fa45daf1c416304ab4680e6af3c1e94b5d4c2657c306","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:47:03Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-d02c78-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-what-does-it-mean-when-an-ai-fails-d02c78-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-d02c78-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e9","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e10","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e11","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:47:03Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-what-does-it-mean-when-an-ai-fails-e6235f-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-f2ac96","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-f2ac96-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-what-does-it-mean-when-an-ai-fails-f2ac96#assessment-marcus-what-does-it-mean-when-an-ai-fails-f2ac96-a1"}],"claim":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-f2ac96","subject_id":"gary-marcus","revision":1,"content_hash":"09dac4a855ae1c079e7ef9f070e775c5eafb94b63e4e3e157bd7b5f10adf017b","normalized_proposition":"Improvement from GPT-2 to GPT-3 has been quantitative only, with zero qualitative change.","kind":"observation","target":"GPT models","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities","scaling"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-what-does-it-mean-when-an-ai-fails-f2ac96-o1","statement_date":"2022-06-07","statement_date_precision":"day","excerpt":"And yes, there has been improvement eg from GPT-2 to GPT-3—but zero qualitative change.","locator":"paragraph 46","document":{"url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","publisher":"Marcus on AI","published_at":"2022-06-07T17:28:09Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:19Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"fa2a39c0800cc5475acd393215c920495373f319a6dd41cc57a65d2d3d491dbc","receipts":1}],"first_statement_date":"2022-06-07","related":[],"current":{"assessment_id":"marcus-what-does-it-mean-when-an-ai-fails-f2ac96-a1","payload":{"claim_id":"marcus-what-does-it-mean-when-an-ai-fails-f2ac96","subject_id":"gary-marcus","normalized_proposition":"Improvement from GPT-2 to GPT-3 has been quantitative only, with zero qualitative change.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"And yes, there has been improvement eg from GPT-2 to GPT-3—but zero qualitative change.","locator":"paragraph 46","url":"https://garymarcus.substack.com/p/what-does-it-mean-when-an-ai-fails","date":"2022-06-07","document_title":"What does it mean when an AI fails? A Reply to SlateStarCodex’s riff on Gary Marcus","observed_at":"2026-10-29T00:05:19Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"fb859ef214eaa645aadfb437f07f26ace0e0820c92b2dac7573d1579ddc91eab","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:56:19Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-what-does-it-mean-when-an-ai-fails-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-what-does-it-mean-when-an-ai-fails-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:56:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-what-does-it-mean-when-an-ai-fails-f2ac96-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-056704","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-056704-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-056704#assessment-marcus-dear-elon-musk-here-are-five-things-056704-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-056704","subject_id":"gary-marcus","revision":1,"content_hash":"45c50e5adf326e44ad090d14e42bcacd0fa4a0fe5b58326ad65dfec79df4e6df","normalized_proposition":"Humanity lacks an adequate methodology for building complex cognitive systems.","kind":"observation","target":"AI engineering","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-056704-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"The fourth thing to realize is that we, and by this I mean humanity as a whole, still don’t really have an adequate methodology for building complex cognitive systems.","locator":"paragraph 27","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-056704-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-056704","subject_id":"gary-marcus","normalized_proposition":"Humanity lacks an adequate methodology for building complex cognitive systems.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"The fourth thing to realize is that we, and by this I mean humanity as a whole, still don’t really have an adequate methodology for building complex cognitive systems.","locator":"paragraph 27","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["humanity as a whole","still don’t really have","an adequate methodology","building complex cognitive systems"],"our_interpretation":["An adequate methodology means a repeatable engineering approach for building and modifying integrated cognitive systems with dependable behavior and manageable regressions.","Adequacy does not require eliminating every bug. Conversely, a successful isolated model does not establish a generally adequate integration methodology.","The context concerns perception, reasoning, planning and action working together, including autonomous vehicles.","The current observation window is calendar 2022 through May 31 because the research field has no regular reporting cycle; earlier methods are context."],"support_condition":"Broad contemporary evidence shows that available engineering approaches cannot reliably manage integration and modification of complex cognitive systems across relevant applications.","contradiction_condition":"A publicly documented, repeatable methodology had demonstrated dependable construction and modification of such systems across relevant applications by May 2022."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The industry study corroborates the specific integration and regression problems Marcus describes. However, concrete testing methods and integrated systems also existed. Persistent failures do not establish that humanity lacked an adequate methodology, just as a successful demonstration does not establish a generally dependable one. The retrieved evidence cannot settle the field-wide claim at the stated level of adequacy.","limitations":["The empirical study covers a sample of practitioners and systems.","The evidence mixes production engineering practices with research demonstrations.","Publication dates do not identify the exact dates of all underlying experiments or interviews."],"evidence":[{"id":"e1","relation":"supports","excerpt":"regression on model performance.","locator":"Abstract, finding 2","url":"https://www.microsoft.com/en-us/research/publication/testing-machine-learning-systems-in-industry-an-empirical-study","publisher":"Microsoft Research; ICSE 2022","event_date":"2022-05","available_at":"2022-05-31","explanation":"The study reports industry integration and regression problems matching Marcus's concern, while also identifying useful practices. The page gives publication month only; available_at uses that month's final day as a conservative availability bound."},{"id":"e2","relation":"context","excerpt":"multi-embodiment generalist policy.","locator":"Abstract","url":"https://arxiv.org/abs/2205.06175v1","publisher":"arXiv; Scott Reed and colleagues","event_date":"2022-05","available_at":"2022-05-12","explanation":"Gato demonstrates integration across several kinds of tasks and embodiments before the post. That is relevant counterevidence to an inability-to-build reading, but it does not demonstrate dependable maintenance or regression control under the contract."},{"id":"e3","relation":"context","excerpt":"technical debt","locator":"Abstract","url":"https://research.google/pubs/the-ml-test-score-a-rubric-for-ml-production-readiness-and-technical-debt-reduction","publisher":"Google Research; Proceedings of IEEE Big Data","event_date":"2017","available_at":"2017-12-31","explanation":"The paper proposes 28 tests and monitoring needs based on production experience, establishing that concrete engineering methods existed. It predates the target period, and the page supplies only a publication year; available_at is a conservative year-end bound."}],"what_would_resolve":"A defined reliability standard and comparative evidence showing whether contemporary methods could repeatedly build, modify and maintain representative integrated cognitive systems to that standard.","retrospective":true,"plain_reading":true},"payload_hash":"cdb08833d1f1f625a1dbdd13fcd73c4a75b21e7b3193ec0d93c0cb24272e1e7d","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T06:55:52Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-056704-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-056704-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-056704-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e3","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T06:55:52Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-056704-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-082354","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-082354-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-082354#assessment-marcus-dear-elon-musk-here-are-five-things-082354-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-082354","subject_id":"gary-marcus","revision":1,"content_hash":"70a967ae0f7aedf0b5199c445ccf34d08b87594523c60561772d499166b73a59","normalized_proposition":"In 2029, AI will not be able to reliably build bug-free code over 10,000 lines from a natural-language spec or from talking with a non-expert user, not counting glued-together library code.","kind":"forecast","target":"AI","conditions":[{"text":"from natural language spec or non-expert interaction","type":"qualifier","from_source":true},{"text":"gluing existing library code doesn't count","type":"qualifier","from_source":true}],"original_horizon_text":"In 2029","topic_tags":["capabilities","agi-timelines"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-082354-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"In 2029, AI will not be able to reliably construct bug-free code of more than 10,000 lines from natural language specification or by interactions with a non-expert user.","locator":"prediction 4 of 5","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-082354-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-082354","subject_id":"gary-marcus","normalized_proposition":"In 2029, AI will not be able to reliably build bug-free code over 10,000 lines from a natural-language spec or from talking with a non-expert user, not counting glued-together library code.","kind":"forecast","conditions":["from natural language spec or non-expert interaction","gluing existing library code doesn't count"],"original_horizon_text":"In 2029","statement":{"excerpt":"In 2029, AI will not be able to reliably construct bug-free code of more than 10,000 lines from natural language specification or by interactions with a non-expert user.","locator":"prediction 4 of 5","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["In 2029","AI will not be able to reliably construct bug-free code","more than 10,000 lines","from natural language specification or by interactions with a non-expert user","Gluing together code from existing libraries doesn’t count."],"our_interpretation":["The deadline is December 31, 2029, allowing the whole named year.","Reliably means repeatable success across varied substantive programming tasks, rather than one selected demonstration. Marcus supplies no numerical success rate.","Bug-free means correctly implementing the requirements without functional defects. Passing a finite test suite is evidence of correctness, but does not by itself establish this.","Count substantive newly generated source code, excluding padding and pre-existing library code. Marcus does not specify a line-count convention.","Either input route qualifies. An expert may supply the natural-language specification under the first route; expert implementation or repair would not establish the AI's capability unaided.","The input restrictions define the task, rather than a conditional forecast requiring a separate antecedent.","This assessment concerns the individual programming prediction, not winning the broader three-of-five AGI bet."],"support_condition":"Evaluations during 2029, covering leading available systems and varied qualifying tasks, consistently find that systems cannot reliably produce correct new programs exceeding 10,000 substantive source lines under either permitted input route. Mere absence of a public demonstration is insufficient.","contradiction_condition":"A system repeatedly produces correct new programs exceeding 10,000 substantive source lines from natural-language specifications or non-expert interaction, with independently assessed correctness and without counting library assembly as the implementation."},"axis":"forecast_outcome","outcome":"unresolved","horizon_state":"not_due","evaluation_as_of":"2026-10-28","rationale":"The deadline is December 31, 2029. The compiler demonstration shows substantial progress beyond the size threshold, but its disclosed defects prevent it from establishing reliable bug-free construction. The verification work proves selected properties of existing software under stated assumptions. Neither meets the early-resolution rule. Present limitations also cannot establish what AI will be unable to do in 2029.","limitations":["The cited demonstrations are developer reports, not independent evaluations satisfying the whole contract.","The search cannot establish that no qualifying system exists.","No numerical reliability threshold was supplied; none is invented here."],"evidence":[{"id":"e1","relation":"context","excerpt":"New features and bugfixes frequently broke existing functionality.","locator":"Stress testing the limits of agent teams, Evaluation","url":"https://www.anthropic.com/engineering/building-c-compiler","publisher":"Anthropic","event_date":"2026-02-05","available_at":"2026-02-05","explanation":"Anthropic reported a newly built 100,000-line compiler, exceeding the forecast's size threshold. The same report describes defects, regressions and substantial expert work on testing, so it does not establish reliable bug-free construction under the complete criteria."},{"id":"e2","relation":"context","excerpt":"“Formally verified” does not mean “bug-free.”","locator":"Limitations","url":"https://www.anthropic.com/research/formal-verification","publisher":"Anthropic","event_date":"2026-10-22","available_at":"2026-10-22","explanation":"The report describes proving security properties for existing software under explicit machine and input assumptions. This is relevant progress, but it does not demonstrate repeated generation of qualifying new programs with complete correctness against natural-language requirements."}],"what_would_resolve":"A reproducible independent evaluation showing repeated qualifying generation could contradict the forecast early. Otherwise, broad evaluations during 2029 must assess the full task, with disclosed success rates, substantive line counts, intervention logs and correctness evidence.","retrospective":false,"plain_reading":true},"payload_hash":"a200f052c8eca1b457d741ce5204270dcfb817e5df036b0c28de8344ed571146","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T06:55:52Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-056704-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-056704-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-056704-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"company_disclosure","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T06:55:52Z","next_review_due_at":null,"horizon_now":"not_due"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-082354-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-12f5f4","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-12f5f4-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-12f5f4#assessment-marcus-dear-elon-musk-here-are-five-things-12f5f4-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-12f5f4","subject_id":"gary-marcus","revision":2,"content_hash":"a335b6bb94ca93e3a5d2aeb265f79c4b0c2a304353a22a2b14d8bbe3e965602d","normalized_proposition":"Many people continue to hope deep learning will be a panacea, which seems pretty unrealistic to Marcus, and he still thinks a whole lot more techniques are needed.","kind":"normative","target":"deep learning","conditions":[{"text":"seems pretty unrealistic to me","type":"qualifier","from_source":true},{"text":"I still think","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities","agi-timelines"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-12f5f4-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"Four years later, many people continue to hope that deep learning will be a panacea; that still seems pretty unrealistic to me; I still think we need a whole lot more techniques.","locator":"paragraph 25","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-12f5f4-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-12f5f4","subject_id":"gary-marcus","normalized_proposition":"Many people continue to hope deep learning will be a panacea, which seems pretty unrealistic to Marcus, and he still thinks a whole lot more techniques are needed.","kind":"normative","conditions":["seems pretty unrealistic to me","I still think"],"original_horizon_text":null,"statement":{"excerpt":"Four years later, many people continue to hope that deep learning will be a panacea; that still seems pretty unrealistic to me; I still think we need a whole lot more techniques.","locator":"paragraph 25","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["seems pretty unrealistic to me","I still think"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found that evidence could not settle it: one found it too vague to test: informed readers would test it in different ways and reach different answers; the other found it a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading","value_judgment"]},"payload_hash":"a0e4fb1f30e96f7221cf6c2905be357ec254040374740a8c9a13fff33e646544","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-12f5f4-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-1663f2","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-1663f2-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-1663f2#assessment-marcus-dear-elon-musk-here-are-five-things-1663f2-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-1663f2","subject_id":"gary-marcus","revision":2,"content_hash":"aa6474244fcf5affa8770dd63f6bd6c5be7975b9f7a8653a106610cea2ecceff","normalized_proposition":"Musk said in 2015 that truly self-driving cars were two years away and has pretty much said the same thing every year since, and truly self-driving cars still had not arrived.","kind":"observation","target":"truly self-driving cars","conditions":[{"text":"pretty much every year since","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities","agi-timelines"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-1663f2-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"You said, for instance in 2015, that (truly) self-driving cars were two years away; you’ve pretty much said the same thing every year since. It still hasn’t happened.","locator":"paragraph 9","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-1663f2-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-1663f2","subject_id":"gary-marcus","normalized_proposition":"Musk said in 2015 that truly self-driving cars were two years away and has pretty much said the same thing every year since, and truly self-driving cars still had not arrived.","kind":"observation","conditions":["pretty much every year since"],"original_horizon_text":null,"statement":{"excerpt":"You said, for instance in 2015, that (truly) self-driving cars were two years away; you’ve pretty much said the same thing every year since. It still hasn’t happened.","locator":"paragraph 9","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["in 2015","(truly) self-driving cars were two years away","pretty much said the same thing every year since","It still hasn’t happened"],"our_interpretation":["The surrounding discussion makes Tesla the ordinary scope of the claim.","Truly self-driving means driving without a human continuously supervising and being ready to intervene.","Pretty much the same thing means repeated near-term forecasts, usually within about two years, rather than an identical two-year promise every calendar year.","The target period covers the explicitly named prediction history through the post date. The non-arrival assertion concerns the end of that period."],"support_condition":"Records establish the 2015 two-year prediction, substantially annual repetitions of near-term autonomy predictions, and Tesla's continued reliance on supervised driving assistance before the post.","contradiction_condition":"Records disprove the attributed 2015 statement or the recurring prediction pattern, or establish that Tesla had delivered the promised unsupervised driving capability before the post."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The 2015 quotation is accurate. Dated accounts document recurring near-term forecasts through 2022, while contemporary testing still required human supervision. This supports the ordinary Tesla-specific reading. The timelines and promised milestones varied, so the evidence does not establish an identical two-year promise every year. Waymo's earlier driverless service means the statement should not be generalized to every company.","limitations":["Several predictions are documented through contemporary reporting because readable original transcripts were unavailable.","The direct 2022 driving observations cover January and February rather than May 31 itself.","The outcome depends on the contextual Tesla-specific scope."],"evidence":[{"id":"e1","relation":"supports","excerpt":"“We’re going to end up with complete autonomy, and I think we will have complete autonomy in approximately two years.”","locator":"The Path to Full Autonomy","url":"https://fortune.com/2015/12/21/elon-musk-interview","publisher":"Fortune","event_date":"2015-12","available_at":"2015-12-21","explanation":"The original interview establishes the attributed prediction. Musk separately allowed additional time for regulatory approval, so the prediction concerned technical capability first."},{"id":"e2","relation":"supports","excerpt":"I feel very confident predicting autonomous robo taxis for Tesla by next year.","locator":"Opening account of Autonomy Day; subsequent dated chronology","url":"https://www.inverse.com/science/55141-elon-musk-autonomous-driving","publisher":"Inverse","event_date":"2019-04-22","available_at":"2019-04-22","explanation":"Reports the 2019 robotaxi forecast. Its chronology also documents a 2016 coast-to-coast forecast, a 2017 two-year sleeping-at-the-wheel forecast, and a 2018 forecast of full self-driving the following year."},{"id":"e3","relation":"supports","excerpt":"Tesla will be able to make its vehicles completely autonomous by the end of this year, founder Elon Musk has said.","locator":"Opening paragraph","url":"https://www.bbc.com/news/technology-53349313","publisher":"BBC News","event_date":"2020-07","available_at":"2020-07-09","explanation":"Documents another near-term prediction in 2020, explicitly concerning autonomy without driver input."},{"id":"e4","relation":"supports","excerpt":"“I’m confident based on my understanding of the technical roadmap and the progress that we’re making between each beta iteration,” Musk said.","locator":"Opening two paragraphs","url":"https://arstechnica.com/cars/2021/05/tesla-autopilot-director-contradicts-musks-self-driving-timeline","publisher":"Ars Technica","event_date":"2021-01","available_at":"2021-05-07","explanation":"The preceding paragraph identifies this as Musk's answer about achieving Level 5 by the end of 2021. This extends the recurring short-timeline pattern into 2021."},{"id":"e5","relation":"supports","excerpt":"All three drivers understood that the technology does not make their Tesla electric cars fully self driving, despite the brand name.","locator":"Introduction describing January and February ride-alongs","url":"https://www.cnbc.com/2022/02/15/tesla-fsd-beta-users-show-cnbc-how-the-system-works-and-doesnt.html","publisher":"CNBC","event_date":"2022-02","available_at":"2022-02-15","explanation":"Direct ride-alongs documented continuing supervision and interventions. The article also reports Musk's renewed prediction, on the 2021 year-end earnings call, of achieving full self-driving during 2022."},{"id":"e6","relation":"context","excerpt":"Beginning today, October 8, we’re excited to open up our fully driverless offering to Waymo One riders.","locator":"Public opening announcement","url":"https://waymo.com/blog/2020/10/waymo-is-opening-its-fully-driverless-service-in-phoenix","publisher":"Waymo","event_date":"2020-10-08","available_at":"2020-10-08","explanation":"Driverless service existed at another company within a limited service area. This is material to the wording's scope, but does not establish that Tesla fulfilled Musk's predictions."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"f5350ba7175a7e7351b2244fa5a2e36d6722e7880a4b991e7b8bd50a6b3a40cd","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T06:55:52Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-056704-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-056704-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-056704-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"news_report","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"news_report","audited":null,"derived":null,"metric":null},{"id":"e3","source_type":"news_report","audited":null,"derived":null,"metric":null},{"id":"e4","source_type":"news_report","audited":null,"derived":null,"metric":null},{"id":"e5","source_type":"news_report","audited":null,"derived":null,"metric":null},{"id":"e6","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T06:55:52Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-1663f2-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-1c4f0d","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-1c4f0d-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-1c4f0d#assessment-marcus-dear-elon-musk-here-are-five-things-1c4f0d-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-1c4f0d","subject_id":"gary-marcus","revision":1,"content_hash":"e45a183695a52a5b0b259a383861d2ec6f8437619e273fd1362f7ffcb3b4c493","normalized_proposition":"Unless the field invests in ideas beyond current techniques like deep learning, outliers alone may prevent reaching AGI before the end of the decade.","kind":"conditional_forecast","target":"the AI field","conditions":[{"text":"unless the field invests in ideas beyond deep learning","type":"antecedent","from_source":true}],"original_horizon_text":"before the end of the decade","topic_tags":["agi-timelines","reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-1c4f0d-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"Seven years is a long time, but the field is going to need to invest in other ideas if we are going to get to AGI before the end of the decade. Or else outliers alone might be enough to keep us from getting there.","locator":"paragraph 17","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-1c4f0d-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-1c4f0d","subject_id":"gary-marcus","normalized_proposition":"Unless the field invests in ideas beyond current techniques like deep learning, outliers alone may prevent reaching AGI before the end of the decade.","kind":"conditional_forecast","conditions":["unless the field invests in ideas beyond deep learning"],"original_horizon_text":"before the end of the decade","statement":{"excerpt":"Seven years is a long time, but the field is going to need to invest in other ideas if we are going to get to AGI before the end of the decade. Or else outliers alone might be enough to keep us from getting there.","locator":"paragraph 17","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["unless the field invests in ideas beyond deep learning","before the end of the decade"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"f25705cbfa26f7dc3058b5b9ef1c2f4c146722358cd7dd8ef24a02e9df83f60f","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:10:54Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:10:54Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-1c4f0d-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-2a322d","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-2a322d-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-2a322d#assessment-marcus-dear-elon-musk-here-are-five-things-2a322d-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-2a322d","subject_id":"gary-marcus","revision":2,"content_hash":"cf39669098791bbd2823ee296b1cc635551d67eb66aab23f56973423b2fa0b00","normalized_proposition":"Unexpected circumstances have been and continue to be the bane of contemporary AI techniques, and probably will remain so until there is a real revolution.","kind":"forecast","target":"contemporary AI techniques","conditions":[{"text":"until there is a real revolution","type":"qualifier","from_source":true},{"text":"probably","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-2a322d-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"Unexpected circumstances have been and continue to be the bane of contemporary AI techniques, and probably will be until there is a real revolution.","locator":"paragraph 16","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-2a322d-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-2a322d","subject_id":"gary-marcus","normalized_proposition":"Unexpected circumstances have been and continue to be the bane of contemporary AI techniques, and probably will remain so until there is a real revolution.","kind":"forecast","conditions":["until there is a real revolution","probably"],"original_horizon_text":null,"statement":{"excerpt":"Unexpected circumstances have been and continue to be the bane of contemporary AI techniques, and probably will be until there is a real revolution.","locator":"paragraph 16","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["until there is a real revolution","probably"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"484ca8b6683af4cf85ce9288aeec2b87fb6aa20cc4d763276e200d4880b140f1","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-2a322d-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-2c8978","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-2c8978-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-2c8978#assessment-marcus-dear-elon-musk-here-are-five-things-2c8978-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-2c8978","subject_id":"gary-marcus","revision":2,"content_hash":"aa8cb29f940f07b70bf5967ec9689abe3f0c01250a82ce544668581224d29026","normalized_proposition":"The AGI needed for a general-purpose domestic robot, where every home is different and poses its own safety risks, is way beyond what is needed for a car driving on roads engineered more or less the same from town to town.","kind":"observation","target":"general-purpose domestic robots (Tesla Optimus)","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities","agi-timelines"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-2c8978-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"I don’t know what you have planned for Optimus, but I can promise you that the AGI that you would need for a general-purpose domestic robot (where every home is different, and each poses its own safety risks) is way beyond what you would need for a car that drives on roads that are more or less engineered the same way from one town to the next.","locator":"paragraph 26","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-2c8978-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-2c8978","subject_id":"gary-marcus","normalized_proposition":"The AGI needed for a general-purpose domestic robot, where every home is different and poses its own safety risks, is way beyond what is needed for a car driving on roads engineered more or less the same from town to town.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"I don’t know what you have planned for Optimus, but I can promise you that the AGI that you would need for a general-purpose domestic robot (where every home is different, and each poses its own safety risks) is way beyond what you would need for a car that drives on roads that are more or less engineered the same way from one town to the next.","locator":"paragraph 26","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["general-purpose domestic robot","every home is different","each poses its own safety risks","way beyond","a car that drives on roads"],"our_interpretation":["The ordinary reading is that safe, general domestic assistance requires substantially broader adaptable cognitive capabilities than autonomous road driving.","This compares required capabilities, not market adoption, commercial success or release dates.","The claim describes the state of knowledge at publication. With no reporting cycle for this research field, the current observation window is calendar 2022 through May 31; earlier studies are context.","The driving comparator is autonomous road driving as described, not merely one fixed route or a specially restricted demonstration."],"support_condition":"Comparative task analyses and capability evaluations establish substantially broader cognitive requirements for safe general domestic work than for autonomous road driving.","contradiction_condition":"Comparable analyses or demonstrations show that general domestic work needs no substantially broader cognitive capabilities, or that driving requires greater generality under comparable assumptions."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The sources make the comparison plausible: household assistance spans many activities, while driving skills can transfer between locations. They also show meaningful household-robot progress and substantial complexity in driving. None directly establishes that the intelligence required for general domestic work is 'way beyond' that required for road driving. Deployment differences would be an insufficient substitute for that comparison.","limitations":["Household robotics evidence includes simulation and constrained kitchen tasks.","Waymo's account is a company disclosure, not an independent comparison with domestic robotics.","Task breadth and overall difficulty are related but distinct."],"evidence":[{"id":"e1","relation":"context","excerpt":"demand generalizable solutions.","locator":"Section 3, Diversity in BEHAVIOR Activities","url":"https://arxiv.org/html/2108.03332v1","publisher":"arXiv; Sanjana Srivastava and colleagues","event_date":"2021","available_at":"2021-08-06","explanation":"The benchmark spans varied household activities, objects and environments, supporting the premise of broad domestic requirements. Its experiments predate the current observation window and do not directly compare cognitive requirements with driving."},{"id":"e2","relation":"context","excerpt":"improves their robustness","locator":"Section 7, Conclusions, Limitations and Future Work","url":"https://arxiv.org/html/2204.01691v1","publisher":"arXiv; Michael Ahn and colleagues","event_date":"2022-04","available_at":"2022-04-04","explanation":"SayCan demonstrates language-guided sequences of real kitchen tasks while identifying limited skills and failure recovery as obstacles. It shows both progress and remaining difficulty, but does not establish the comparative amount of intelligence required."},{"id":"e3","relation":"context","excerpt":"fundamentals of driving","locator":"Section Using Machine Learning to Build a Generalizable System","url":"https://waymo.com/blog/2022/05/howwevebuiltascalableautonomousdriver","publisher":"Waymo","event_date":"2022-05","available_at":"2022-05-18","explanation":"Waymo describes transferable driving skills and autonomous rides, consistent with Marcus's premise of shared road-driving requirements. The same disclosure describes rare events, local rules and climate differences, limiting any inference that driving environments are simple or uniform."}],"what_would_resolve":"A comparative requirements analysis or controlled capability study addressing general domestic work and autonomous driving with explicit operating scope and comparable reliability requirements.","retrospective":true,"plain_reading":true},"payload_hash":"4f07eed9ae188c8bc238caaa2d347700f19ce765c1b66c0870e801619a985d79","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T06:55:52Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-056704-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-056704-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-056704-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e3","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T06:55:52Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-2c8978-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-31b9db","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-31b9db-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-31b9db#assessment-marcus-dear-elon-musk-here-are-five-things-31b9db-a2"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-31b9db","subject_id":"gary-marcus","revision":2,"content_hash":"a7a43d09b5d8179e23fe50996567894a36bc2bafb1eb7c2eb7c39f40d5468025","normalized_proposition":"In 2029, AI will not be able to read a novel and reliably answer questions about plot, character, conflicts, motivations, etc., where the key is going beyond the literal text.","kind":"forecast","target":"AI","conditions":[{"text":"key will be going beyond the literal text","type":"qualifier","from_source":true}],"original_horizon_text":"In 2029","topic_tags":["capabilities","agi-timelines"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-31b9db-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"In 2029, AI will not be able to read a novel and reliably answer questions about plot, character, conflicts, motivations, etc.","locator":"prediction 2 of 5","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-31b9db-a2","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-31b9db","subject_id":"gary-marcus","normalized_proposition":"In 2029, AI will not be able to read a novel and reliably answer questions about plot, character, conflicts, motivations, etc., where the key is going beyond the literal text.","kind":"forecast","conditions":["key will be going beyond the literal text"],"original_horizon_text":"In 2029","statement":{"excerpt":"In 2029, AI will not be able to read a novel and reliably answer questions about plot, character, conflicts, motivations, etc.","locator":"prediction 2 of 5","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["In 2029","read a novel","reliably answer questions about plot, character, conflicts, motivations, etc.","going beyond the literal text"],"our_interpretation":["The ordinary reading is dependable comprehension of whole novels, including reasonable inferences about motives, relationships and conflicts.","Going beyond the literal text defines the task. It is not an external antecedent whose occurrence must be established separately.","Test varied novels and questions with controls for memorized summaries. Reliability does not require agreement on every debatable literary interpretation.","Use 2029-12-31 as the latest deadline, allowing an earlier reproducible demonstration of the complete capability to refute the forecast.","Assess this individual forecast independently of the surrounding multi-challenge wager."],"support_condition":"Evidence covering the end of 2029 shows that leading available systems still fail to answer questions dependably about whole novels, including questions requiring inference about plot, characters, conflicts and motivations.","contradiction_condition":"At least one AI system reliably answers questions across varied whole novels, including nonliteral questions, under a credible evaluation with human-validated answers."},"axis":"forecast_outcome","outcome":"unresolved","horizon_state":"not_due","evaluation_as_of":"2026-10-28","rationale":"The August 2025 PRELUDE study reports that tested systems lagged behind humans when judging whether character backstories fit a novel [e1]. The September 2026 detective-novel study tests one local 9B model on multiple-choice questions and reports limited accuracy [e2]. Neither establishes the complete, independently assessed and reproducible whole-novel capability required for early contradiction, and these earlier results cannot establish failure in 2029 [e1; e2].","limitations":["The studies do not establish current frontier performance across the complete task or what systems will achieve in 2029.","The packet contains no verified Fiction.LiveBench results; their relevance to early resolution remains unassessed.","The claim supplies no numerical reliability threshold, and literary questions can admit multiple defensible answers."],"evidence":[{"id":"e1","relation":"context","excerpt":"room for improvement","locator":"Abstract, final sentence","url":"https://arxiv.org/abs/2508.09848v2","publisher":"arXiv; Mo Yu and collaborators","event_date":"2025","available_at":"2025-08-14","explanation":"PRELUDE tests whether proposed character backstories fit a novel, requiring indirect evidence and inference. Its reported human-model gaps concern a demanding related task and do not settle the broader 2029 forecast."},{"id":"e2","relation":"context","excerpt":"the graph route reaches 42.86%.","locator":"Abstract, results discussion","url":"https://arxiv.org/abs/2609.22939","publisher":"arXiv; Wenji Fu","event_date":"2026-09","available_at":"2026-09-19","explanation":"This study evaluates a fixed local model on detective-novel questions and reports limited performance. It neither establishes general reliable comprehension nor measures the strongest available systems."}],"what_would_resolve":"A dated, independently assessed and reproducible evaluation demonstrating reliable answers across varied whole novels, including nonliteral questions, would contradict the forecast early. It must supply the novels, disclose outputs and grading, use human-validated answers, accept text-supported alternative interpretations, and control for prior familiarity and answer leakage. Otherwise, evidence covering leading systems through December 31, 2029 would be needed to assess whether dependable comprehension remains beyond them.","retrospective":false,"plain_reading":true},"payload_hash":"d20454339b0bea4567351aff39ce1801ed0d45f618aa44ca8a083e5d43bc7909","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T07:09:09Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-31b9db-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-31b9db-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-31b9db-2x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T07:09:09Z","next_review_due_at":null,"horizon_now":"not_due"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-31b9db-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-536f05","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-536f05-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-536f05#assessment-marcus-dear-elon-musk-here-are-five-things-536f05-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-536f05","subject_id":"gary-marcus","revision":1,"content_hash":"904472ff23993c9299e4a8224f7574865f5b60fb630d229d48e335a4647df2fe","normalized_proposition":"AI has no stable or trustworthy solutions for common sense, reasoning, language or analogy, and this has scarcely changed in about five years.","kind":"observation","target":"current AI","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities","reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-536f05-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"We still don’t have anything like stable or trustworthy solutions for common sense, reasoning, language, or analogy. Truth is, I have been using this same pie chart for about 5 years, and the situation has scarcely changed.","locator":"paragraph 21","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-536f05-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-536f05","subject_id":"gary-marcus","normalized_proposition":"AI has no stable or trustworthy solutions for common sense, reasoning, language or analogy, and this has scarcely changed in about five years.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"We still don’t have anything like stable or trustworthy solutions for common sense, reasoning, language, or analogy. Truth is, I have been using this same pie chart for about 5 years, and the situation has scarcely changed.","locator":"paragraph 21","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["stable or trustworthy solutions","common sense, reasoning, language, or analogy","about 5 years","the situation has scarcely changed"],"our_interpretation":["A trustworthy solution means dependable performance across varied, unfamiliar instances of a domain, rather than success on one benchmark. It does not require perfect performance.","The comparison covers approximately 2017 through May 2022. Calendar 2017 is the approximate baseline; the endpoint is the post date.","In context, 'scarcely changed' concerns the availability of dependable general solutions. It does not necessarily deny substantial improvements in benchmark scores."],"support_condition":"Contemporary evaluations show persistent, substantial reliability limitations across the four domains, and comparable longitudinal evidence shows little change in the availability of dependable general solutions.","contradiction_condition":"A dependable general solution existed in a named domain by May 2022, or comparable evaluations demonstrate substantial improvement in broad reliability since approximately 2017."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Contemporary research supports persistent weaknesses in language, commonsense reasoning and analogy. PaLM also documents substantial benchmark progress before the post. Those findings can coexist: improved scores do not establish trustworthy general solutions. The evidence does not settle the combined claim that dependable solutions were absent across all four domains and that this situation had scarcely changed over five years.","limitations":["The studies use different tasks, models and evaluation methods.","Scoring the claim as mixed solely because benchmark scores improved would substitute a different meaning of progress."],"evidence":[{"id":"e1","relation":"supports","excerpt":"perform poorly under adversarial evaluation.","locator":"Section 5, Discussion and Related Work","url":"https://arxiv.org/html/1707.07328v1","publisher":"arXiv; Robin Jia and Percy Liang","event_date":"2017","available_at":"2017-07-23","explanation":"The experiments establish substantial reading-comprehension brittleness near the five-year baseline. They do not measure subsequent improvement."},{"id":"e2","relation":"supports","excerpt":"word embedding models.","locator":"Abstract","url":"https://arxiv.org/abs/2105.04949v1","publisher":"arXiv; Asahi Ushio and colleagues","event_date":"2021","available_at":"2021-05-11","explanation":"The study reports difficulty with abstract analogies and sensitivity to architecture and settings. This supports a limited finding of unreliable analogy performance among the tested models, not the absence of every possible solution in May 2022."},{"id":"e3","relation":"supports","excerpt":"partially understood","locator":"Discussion, Exploring new components of commonsense knowledge","url":"https://arxiv.org/html/2201.12438v1","publisher":"arXiv; Prajjwal Bhargava and Vincent Ng","event_date":"2022-01","available_at":"2022-01-28","explanation":"This contemporary survey identifies unaddressed commonsense capabilities and weaknesses in existing evaluations. It supports continued incompleteness but supplies no common five-year measure of reliability."},{"id":"e4","relation":"context","excerpt":"language understanding and generation benchmarks.","locator":"Abstract","url":"https://arxiv.org/abs/2204.02311v1","publisher":"arXiv; Aakanksha Chowdhery and colleagues","event_date":"2022-04","available_at":"2022-04-05","explanation":"PaLM supplies material counterevidence to a reading that little capability progress occurred. These benchmark gains do not by themselves establish dependable general solutions under the contract."}],"what_would_resolve":"Comparable historical robustness evaluations across the four domains, together with a defensible threshold for dependable performance and meaningful five-year change.","retrospective":true,"plain_reading":true},"payload_hash":"3ee8bb0ead86ed2168d2ad0642f8cae62bc3aa16300b14b0a22b5ce884ad64b9","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T06:55:56Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-31b9db-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-31b9db-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-31b9db-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e3","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e4","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T06:55:56Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-536f05-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-5dc69d","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-5dc69d#assessment-marcus-dear-elon-musk-here-are-five-things-5dc69d-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d","subject_id":"gary-marcus","revision":1,"content_hash":"1f7efb5811b3455c2c1bdf3235c768ef4ba3d38aef9ea59260ee156cc5868046","normalized_proposition":"Deep learning is decent at object recognition but much worse at planning, reading and language comprehension.","kind":"observation","target":"deep learning","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-5dc69d-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"Deep learning is pretty decent at, for example, recognizing objects, but not nearly as good at planning, or reading, or language comprehension.","locator":"paragraph 20","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d","subject_id":"gary-marcus","normalized_proposition":"Deep learning is decent at object recognition but much worse at planning, reading and language comprehension.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Deep learning is pretty decent at, for example, recognizing objects, but not nearly as good at planning, or reading, or language comprehension.","locator":"paragraph 20","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["Deep learning","pretty decent at","recognizing objects","not nearly as good at planning, or reading, or language comprehension"],"our_interpretation":["The ordinary reading compares the maturity and reliability of deep learning across the named abilities, rather than comparing unrelated benchmark percentages.","Reading is understood as understanding written material, given its pairing with language comprehension.","Human-relative task reliability is a useful comparison standard, but Marcus supplies neither this measure nor a required size of gap.","The observation concerns the current 2022 research record through May 31; academic research has no fixed reporting quarter."],"support_condition":"Representative, reasonably comparable evaluations establish useful object recognition and substantially weaker reliability in planning, reading, and language comprehension.","contradiction_condition":"Representative, reasonably comparable evaluations establish that the claimed broad disadvantage in the other abilities does not hold."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The evidence establishes strong object recognition and real planning limitations. It also establishes strong language benchmark performance before the post. These findings do not settle the comparative claim: the tasks, metrics, and evaluation conditions differ. Calling the whole statement supported or contradicted would depend too heavily on which tasks the evaluator selected.","limitations":["Benchmark success does not establish broad real-world comprehension.","A planning failure does not by itself establish that planning was substantially worse than recognition.","The evidence supports individual observations without resolving the claimed ranking."],"evidence":[{"id":"e1","relation":"context","excerpt":"91.0% on ImageNet","locator":"Section 1, Introduction, page 2","url":"https://arxiv.org/pdf/2205.01917v1","publisher":"arXiv; Jiahui Yu and colleagues","event_date":"2022-05-04","available_at":"2022-05-04","explanation":"This supports the component claim that deep learning was effective at object recognition. The different task scores do not establish a common scale for comparing recognition with planning or comprehension."},{"id":"e2","relation":"context","excerpt":"admissible actions","locator":"Abstract","url":"https://arxiv.org/abs/2201.07207v1","publisher":"arXiv; Wenlong Huang and colleagues","event_date":"2022-01-18","available_at":"2022-01-18","explanation":"The study documents a practical planning limitation while also finding useful planning ability and improvements from translating plans into admissible actions. It does not compare planning reliability with object recognition."},{"id":"e3","relation":"context","excerpt":"language understanding","locator":"Abstract","url":"https://arxiv.org/abs/2204.02311v1","publisher":"arXiv; Aakanksha Chowdhery and colleagues","event_date":"2022-04-05","available_at":"2022-04-05","explanation":"PaLM reports strong language results, including performance above the average human result on its BIG-bench evaluation. This challenges a blanket characterization of weak language ability but does not supply the required comparison with object recognition."}],"what_would_resolve":"A representative evaluation of systems available by May 2022 using a defensible common standard across all four abilities, with an explicit treatment of task difficulty and generalization.","retrospective":true,"plain_reading":true},"payload_hash":"20800551b7adca7e351869cfea55ca3f86eaa660d8274b8afc7a675a32d2e2b7","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T06:56:00Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e3","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T06:56:00Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-5dc69d-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-720a9f","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-720a9f-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-720a9f#assessment-marcus-dear-elon-musk-here-are-five-things-720a9f-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-720a9f","subject_id":"gary-marcus","revision":2,"content_hash":"0beaa7fcdd33ecfe7f808a708664406fa65964d36531554eef2afe46791f8d96","normalized_proposition":"In 2029, AI will not be able to take arbitrary proofs written in natural language from the mathematical literature and convert them into a symbolic form suitable for symbolic verification.","kind":"forecast","target":"AI","conditions":[{"text":"from the mathematical literature","type":"scope","from_source":true}],"original_horizon_text":"In 2029","topic_tags":["capabilities","agi-timelines"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-720a9f-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"In 2029, AI will not be able to take arbitrary proofs from the mathematical literature written in natural language and convert them into a symbolic form suitable for symbolic verification.","locator":"prediction 5 of 5","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-720a9f-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-720a9f","subject_id":"gary-marcus","normalized_proposition":"In 2029, AI will not be able to take arbitrary proofs written in natural language from the mathematical literature and convert them into a symbolic form suitable for symbolic verification.","kind":"forecast","conditions":["from the mathematical literature"],"original_horizon_text":"In 2029","statement":{"excerpt":"In 2029, AI will not be able to take arbitrary proofs from the mathematical literature written in natural language and convert them into a symbolic form suitable for symbolic verification.","locator":"prediction 5 of 5","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["In 2029","AI will not be able to take arbitrary proofs","from the mathematical literature","written in natural language","convert them into a symbolic form suitable for symbolic verification"],"our_interpretation":["The deadline is December 31, 2029, allowing the whole named year.","Arbitrary means broad applicability to independently selected proofs across mathematical literature, rather than selected showcases or one narrow class. It does not require successfully verifying false arguments.","Converting proofs means preserving their mathematical content and substantive reasoning while supplying formal details. Producing a different proof of the same theorem alone does not demonstrate faithful conversion.","Successful checking by a proof assistant is the practical test of suitability for symbolic verification. Statements alone, unchecked proof holes and unsupported assumptions do not establish success.","Marcus specifies neither a representative sampling method nor an acceptable failure rate; those must be disclosed rather than silently invented.","The literature restriction defines the task, rather than a conditional forecast requiring a separate antecedent.","This assessment concerns the individual formalization prediction, not winning the broader three-of-five AGI bet."],"support_condition":"Broad evaluations during 2029 find that leading systems still cannot faithfully formalize independently selected natural-language proofs from diverse mathematical literature, with recurring substantive failures rather than merely occasional errors. Absence of a demonstration alone is insufficient.","contradiction_condition":"Reproducible evaluations establish broad successful conversion of independently selected literature proofs into checked symbolic proofs, preserving the original mathematical content and reasoning without substantial expert completion."},"axis":"forecast_outcome","outcome":"unresolved","horizon_state":"not_due","evaluation_as_of":"2026-10-28","rationale":"The deadline is December 31, 2029. Textbook-scale formalization and the reported checked proof of Fermat's Last Theorem are substantial counterevidence to any suggestion that literature-proof formalization is wholly beyond AI. They do not establish the broad arbitrary-proof capability in this forecast. The broader benchmark also supplies human-formalized statements. No reviewed evidence meets the early-resolution rule, and current gaps cannot establish failure in 2029.","limitations":["The research and developer reports were read, but their generated proof artifacts were not independently rerun in this assessment.","Breadth and proof fidelity require judgments the speaker did not fully specify.","A checked proof of a theorem does not alone establish faithful translation of its source argument."],"evidence":[{"id":"e3","relation":"context","excerpt":"large-scale automated formalization of mathematical literature","locator":"Abstract, penultimate sentence","url":"https://arxiv.org/abs/2602.17016","publisher":"arXiv","event_date":"2026-02-19","available_at":"2026-02-19","explanation":"M2F reports 153,853 lines of Lean from 479 textbook pages on real and convex analysis. This is material progress, but those selected corpora and the reported statement audits do not establish faithful conversion of arbitrary literature proofs across mathematics."},{"id":"e4","relation":"context","excerpt":"computer-checked proof of Fermat’s Last Theorem.","locator":"Opening paragraph","url":"https://www.anthropic.com/research/formalizing-fermats-last-theorem","publisher":"Anthropic","event_date":"2026-08-18","available_at":"2026-09-04","explanation":"Anthropic reports a checked formalization following a published exposition of Wiles's proof, with limited high-level human input; its quoted completion log dates the achievement to August 18 UTC. This strongly challenges a reading that AI cannot formalize major literature proofs, but a selected project does not establish the forecast's arbitrary-proof breadth."},{"id":"e5","relation":"context","excerpt":"leaving the corresponding Lean proof","locator":"Abstract, fifth sentence","url":"https://ai.meta.com/research/publications/autoformbench-a-lean-formalization-benchmark-for-advanced-mathematics","publisher":"Meta AI","event_date":"2026-10-13","available_at":"2026-10-13","explanation":"AutoformBench covers 341 tasks from 24 advanced mathematical sources and supplies informal proofs, making it relevant to breadth. Human-formalized target statements provide part of the translation, and the announcement does not establish general faithful conversion of arbitrary literature proofs."}],"what_would_resolve":"An independent, reproducible evaluation across diverse literature proofs, selected without favoring the system, with source-fidelity review, complete checker results and disclosed human interventions. Such evidence could establish capability early; otherwise evaluations during 2029 must settle the forecast.","retrospective":false,"plain_reading":true},"payload_hash":"950bb3d8a76b601128c29aaa431e15a7d374332414f154b8744d2c7b33380169","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T06:56:00Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e3","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e4","source_type":"company_disclosure","audited":null,"derived":null,"metric":null},{"id":"e5","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T06:56:00Z","next_review_due_at":null,"horizon_now":"not_due"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-720a9f-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-8818ae","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-8818ae-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-8818ae#assessment-marcus-dear-elon-musk-here-are-five-things-8818ae-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-8818ae","subject_id":"gary-marcus","revision":1,"content_hash":"239ae72d5c42d78f6f93a08323fb938516e5fd90607e65bf63a0672ac2ad6b8d","normalized_proposition":"3D perception and scene understanding remain unsolved for current AI.","kind":"observation","target":"current AI","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-8818ae-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"Even within perception 3D perception remains a challenge, and scene understanding is by no means solved.","locator":"paragraph 21","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-8818ae-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-8818ae","subject_id":"gary-marcus","normalized_proposition":"3D perception and scene understanding remain unsolved for current AI.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Even within perception 3D perception remains a challenge, and scene understanding is by no means solved.","locator":"paragraph 21","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["Even within perception","3D perception remains a challenge","scene understanding is by no means solved"],"our_interpretation":["The ordinary reading is that substantial technical failures remained in reliable 3D perception and scene understanding.","Solved does not require flawless performance, and the claim does not deny useful systems or progress.","Scene understanding includes identifying objects and their spatial relationships within a scene.","The observation concerns the current 2022 research record through May 31; academic research has no fixed reporting quarter."],"support_condition":"Contemporaneous evaluations document material remaining limitations in both 3D perception and scene understanding.","contradiction_condition":"Contemporaneous evidence establishes broadly reliable solutions in both areas, with remaining errors too limited to sustain the ordinary claim."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Research available before the post documents significant weaknesses in 3D recognition under common corruptions and a remaining object-identification bottleneck in spatial scene question answering. These are substantive limitations in both named areas. The studies also show progress, which is compatible with Marcus's claim that the problems remained challenging and unsolved.","limitations":["The studies cover particular tasks rather than every form of perception or scene understanding.","ScanQA combines perception and language; its limitations cannot all be attributed to perception alone.","The ScanQA work first appeared in 2021; the cited evidence is its publicly available May 2022 revision."],"evidence":[{"id":"e1","relation":"supports","excerpt":"a significant gap","locator":"Abstract","url":"https://arxiv.org/abs/2201.12296v1","publisher":"arXiv; Jiachen Sun and colleagues","event_date":"2022-01-28","available_at":"2022-01-28","explanation":"Testing fifteen common corruptions exposed substantial robustness weaknesses in leading 3D recognition models. The authors also report mitigation methods, so the result establishes remaining challenges rather than an absence of progress."},{"id":"e2","relation":"supports","excerpt":"oracle setting","locator":"Section 5.2, comparison of ScanQA with image-based question-answering baselines","url":"https://arxiv.org/html/2112.10482v3","publisher":"arXiv; Daichi Azuma and colleagues","event_date":"2022-05-07","available_at":"2022-05-07","explanation":"The May revision reports that supplying correct object identification can make an image-based baseline competitive with or better than ScanQA. Its authors identify that assistance as unavailable in real-world use, demonstrating a remaining scene-understanding bottleneck."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"2a48ff2270d1705604bb2b7902283e468eadadf0d83aa1c4dd2760d353d32de9","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T06:56:00Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T06:56:00Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-8818ae-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-a53d89","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-a53d89-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-a53d89#assessment-marcus-dear-elon-musk-here-are-five-things-a53d89-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-a53d89","subject_id":"gary-marcus","revision":2,"content_hash":"3a4b2181b4efda74b150d3e516b9d9940414780159e8b855c2b31ced2dc3bee9","normalized_proposition":"The long-tail problem of rare events is a very serious issue for the currently popular AI techniques, which rely more on big data than on reasoning.","kind":"observation","target":"current big-data AI techniques","conditions":[{"text":"for the kinds of AI techniques that are currently popular","type":"scope","from_source":true}],"original_horizon_text":null,"topic_tags":["reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-a53d89-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"But for the kinds of AI techniques that are currently popular, and which rely more on big data than on reasoning, it’s a very serious issue.","locator":"paragraph 10","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-a53d89-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-a53d89","subject_id":"gary-marcus","normalized_proposition":"The long-tail problem of rare events is a very serious issue for the currently popular AI techniques, which rely more on big data than on reasoning.","kind":"observation","conditions":["for the kinds of AI techniques that are currently popular"],"original_horizon_text":null,"statement":{"excerpt":"But for the kinds of AI techniques that are currently popular, and which rely more on big data than on reasoning, it’s a very serious issue.","locator":"paragraph 10","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["AI techniques that are currently popular","rely more on big data than on reasoning","a very serious issue"],"our_interpretation":["The surrounding text identifies the issue as poor handling of rare cases with little training data.","The ordinary reading concerns mainstream data-driven deep learning, not every AI method or an assertion that neural networks cannot reason at all.","Very serious means substantial performance or reliability limitations, not a claim that the problem is impossible to mitigate.","For this continuously reported technical field, current is operationalized as the 2022 state of the art up to the post date, rather than an accounting quarter."],"support_condition":"Contemporary research documents substantial deficiencies on rare classes or situations in mainstream deep-learning systems, despite stronger performance where training data are plentiful.","contradiction_condition":"Representative contemporary evaluations show that rare-case limitations are generally small or robustly resolved in the relevant methods."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"A research synthesis publicly available four days before the post identifies poor rare-class performance as a substantial limitation of conventional deep learning. That supports the ordinary reading of a serious practical issue. Existing mitigation methods show progress, so the evidence does not support stronger claims of universal failure or impossibility.","limitations":["Evidence is strongest for visual recognition.","Seriousness is qualitative rather than tied to a speaker-specified threshold.","The comparison with reasoning is a description of the targeted methods, not a separately established ranking of all AI approaches."],"evidence":[{"id":"e1","relation":"supports","excerpt":"Using conventional methods (e.g., Cross-Entropy loss, or simple fine-tuning, etc.) will result in the poor performance of the tail classes.","locator":"Section 2.1, discussion of transfer learning","url":"https://ar5iv.labs.arxiv.org/html/2205.13775","publisher":"International Journal of Computer Vision; author manuscript rendered by ar5iv","event_date":"2022-05","available_at":"2022-05-27","explanation":"The contemporary synthesis documents poor learning of rare classes and substantial limitations in conventional deep-learning methods. It also reviews mitigation methods, supporting a serious practical problem without implying that every method fails."},{"id":"e2","relation":"context","excerpt":"Experiments on long-tailed CIFAR, ImageNet, Places, and iNaturalist 2018 manifest the new state-of-the-art for long-tailed recognition.","locator":"Abstract","url":"https://arxiv.org/abs/2107.12028v1","publisher":"arXiv","event_date":"2021-07","available_at":"2021-07-26","explanation":"PaCo demonstrates progress on several long-tailed benchmarks using changes within deep learning. This qualifies any stronger interpretation that the problem had no useful mitigation, but does not show it was generally solved in 2022."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"1d6c76b6910875bb289036929e676670c7438a8b03e4b61f0142c26bd11aa53c","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T06:56:00Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-5dc69d-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T06:56:00Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-a53d89-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-acced4","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-acced4-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-acced4#assessment-marcus-dear-elon-musk-here-are-five-things-acced4-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-acced4","subject_id":"gary-marcus","revision":1,"content_hash":"8b24025cf4d649799ddc9c0766949014604bd7cfcfe5df34aa496821cbca733b","normalized_proposition":"In 2029, AI will not be able to work as a competent cook in an arbitrary kitchen.","kind":"forecast","target":"AI","conditions":[],"original_horizon_text":"In 2029","topic_tags":["capabilities","agi-timelines"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-acced4-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"In 2029, AI will not be able to work as a competent cook in an arbitrary kitchen (extending Steve Wozniak’s cup of coffee benchmark).","locator":"prediction 3 of 5","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-acced4-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-acced4","subject_id":"gary-marcus","normalized_proposition":"In 2029, AI will not be able to work as a competent cook in an arbitrary kitchen.","kind":"forecast","conditions":[],"original_horizon_text":"In 2029","statement":{"excerpt":"In 2029, AI will not be able to work as a competent cook in an arbitrary kitchen (extending Steve Wozniak’s cup of coffee benchmark).","locator":"prediction 3 of 5","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["In 2029","work as a competent cook","in an arbitrary kitchen","extending Steve Wozniak’s cup of coffee benchmark"],"our_interpretation":["The ordinary reading requires an embodied AI system that can prepare ordinary meals competently in unfamiliar, usable kitchens.","'Arbitrary' means varied ordinary kitchens without installation tailored to the robot; it does not mean impossible or unequipped environments.","Competence includes locating and using equipment, handling ingredients, managing cooking stages, and producing safe, edible food with reasonable consistency.","Human meal requests are allowed, but teleoperation or human completion of cooking steps does not demonstrate the robot's competence.","Use 2029-12-31 as the latest deadline and assess this forecast independently of the surrounding wager."],"support_condition":"Evidence covering the end of 2029 shows that available AI robots still cannot consistently complete ordinary cooking tasks in varied unfamiliar kitchens without substantial human assistance or robot-specific installation.","contradiction_condition":"An AI robot repeatedly prepares ordinary meals competently and safely across varied unfamiliar kitchens without teleoperation, task completion by people, or bespoke kitchen installation."},"axis":"forecast_outcome","outcome":"unresolved","horizon_state":"not_due","evaluation_as_of":"2026-10-28","rationale":"The forecast is not due. YORI demonstrates cooking in a structured installation, while Figure reports unfamiliar-home performance on other household tasks. These are relevant advances, but neither establishes competent cooking in arbitrary kitchens. No inspected evidence meets the early-resolution rule.","limitations":["The cooking demonstrations and unfamiliar-home results concern different systems and tasks.","A company report is not independent confirmation.","The absence of a qualifying result in this search does not establish that none exists."],"evidence":[{"id":"e1","relation":"context","excerpt":"reliably preparing steak-frites","locator":"Abstract, sixth sentence","url":"https://arxiv.org/abs/2405.11094v4","publisher":"arXiv; Donghun Noh and collaborators","event_date":"2025","available_at":"2025-11-24","explanation":"YORI provides relevant counterpressure to a broad claim that robots cannot cook. The report specifies custom tools, appliances and a structured kitchen, so it does not demonstrate competence in arbitrary kitchens."},{"id":"e2","relation":"context","excerpt":"tidying a room","locator":"Section The Whole-Body Generalization Problem, first paragraph","url":"https://www.figure.ai/news/helix-2-5-zero-shot-30-home-generalization","publisher":"Figure","event_date":"2026-09","available_at":"2026-09-17","explanation":"Figure reports evaluation across 30 unseen homes, which is relevant to generalization. The evaluated behaviors exclude cooking, so this does not satisfy the forecast's combined task and environment requirements."}],"what_would_resolve":"An independently observed, repeatable evaluation showing autonomous meal preparation across varied unfamiliar kitchens would contradict the forecast early. Otherwise assess the complete capability through the end of 2029.","retrospective":false,"plain_reading":true},"payload_hash":"fb033f83c8127334af5522d5bf03c0524f8d40d45209582280974c5536584b0d","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T06:56:13Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-acced4-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-acced4-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-acced4-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T06:56:13Z","next_review_due_at":null,"horizon_now":"not_due"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-acced4-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-b5e529","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-b5e529-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-b5e529#assessment-marcus-dear-elon-musk-here-are-five-things-b5e529-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-b5e529","subject_id":"gary-marcus","revision":2,"content_hash":"40f0a7ca178b45cbbeff0368ce40bd2cdf26bad520a486c7d84cb378acdef944","normalized_proposition":"Seven years might well not be enough to invent the further techniques needed, if they do not already exist, and to take them from the lab into production.","kind":"forecast","target":"the AI field","conditions":[{"text":"if they don't already exist","type":"qualifier","from_source":true},{"text":"being realistic","type":"qualifier","from_source":true}],"original_horizon_text":"7 years","topic_tags":["agi-timelines"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-b5e529-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"Being realistic, 7 years might well not be enough to (a) invent those tools (if they are don’t already exist) and (b) take them from the lab into production.","locator":"paragraph 25","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-b5e529-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-b5e529","subject_id":"gary-marcus","normalized_proposition":"Seven years might well not be enough to invent the further techniques needed, if they do not already exist, and to take them from the lab into production.","kind":"forecast","conditions":["if they don't already exist","being realistic"],"original_horizon_text":"7 years","statement":{"excerpt":"Being realistic, 7 years might well not be enough to (a) invent those tools (if they are don’t already exist) and (b) take them from the lab into production.","locator":"paragraph 25","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["if they don't already exist","being realistic","7 years"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"e808aec6a7248a1457bc60751b595d55536b402c98b3ff44e1202f4a4f05545f","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-b5e529-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-d1f821","standing":"current","assessment_label":"Mixed","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-d1f821-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-d1f821#assessment-marcus-dear-elon-musk-here-are-five-things-d1f821-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-d1f821","subject_id":"gary-marcus","revision":2,"content_hash":"63f0fc406a4670dbad93f488de9833a508771d8870bcd0216fc136933981fbf9","normalized_proposition":"Debugging deep learning is wicked hard because nobody really understands how it works or knows how to fix problems other than collecting more data, adding more layers and so forth.","kind":"causal","target":"deep learning","conditions":[],"original_horizon_text":null,"topic_tags":["reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-d1f821-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"Debugging with deep learning is wicked hard, because nobody (a) really understands how it works, and (b) knows how to fix problems other to collect more data and add more layers and so forth.","locator":"paragraph 29","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-d1f821-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-d1f821","subject_id":"gary-marcus","normalized_proposition":"Debugging deep learning is wicked hard because nobody really understands how it works or knows how to fix problems other than collecting more data, adding more layers and so forth.","kind":"causal","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Debugging with deep learning is wicked hard, because nobody (a) really understands how it works, and (b) knows how to fix problems other to collect more data and add more layers and so forth.","locator":"paragraph 29","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["Debugging with deep learning is wicked hard","nobody (a) really understands how it works","knows how to fix problems other to collect more data and add more layers and so forth"],"our_interpretation":["Assess the state of knowledge available when the post appeared, on 2022-05-31. The target period has no lower bound because established earlier methods remain relevant to that state.","The ordinary reading is that limited understanding makes diagnosis and repair difficult, and that practitioners lack effective targeted remedies beyond broadly changing data, training, or model size.","Assess the difficulty and its proposed explanation separately. A targeted repair need not solve debugging generally to count against the assertion that such remedies were unknown.","Do not interpret 'understands' as requiring complete mathematical understanding, or 'and so forth' as automatically encompassing every possible repair."],"support_condition":"Contemporaneous research or engineering evidence shows that model opacity materially obstructed diagnosis and repair, and that effective remedies were generally limited to broad data or training changes.","contradiction_condition":"Contemporaneous work demonstrates useful understanding of internal mechanisms and effective targeted repairs beyond collecting more data or enlarging models. Evidence that debugging was routinely straightforward would also contradict the difficulty claim."},"axis":"causal_assessment","outcome":"mixed","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Contemporary research supports Marcus's point that opacity makes debugging harder. But the stronger explanation overstates the limits of existing methods: before the post, ROME demonstrated causal investigation of factual recall and targeted edits to model weights. The difficulty claim holds in this narrower sense; the assertion that useful understanding and targeted remedies were absent does not.","limitations":["The evidence concerns neural language models, not every deep learning application.","The supporting survey gives a qualitative assessment, not a controlled estimate of the causal contribution of opacity.","The mixed verdict depends on treating the assertion about absent remedies as a substantive commitment rather than loose rhetoric."],"evidence":[{"id":"e1","relation":"supports","excerpt":"opaque deep learning models.","locator":"Abstract, second sentence","url":"https://arxiv.org/abs/2104.15135v1","publisher":"arXiv; Piyawat Lertvittayakumjorn and Francesca Toni","event_date":"2021-04","available_at":"2021-04-30","explanation":"The survey supports the narrower causal point that opacity makes locating and repairing problems harder. Its review of explanation-based debugging also shows that researchers were already developing remedies."},{"id":"e2","relation":"contradicts","excerpt":"editing facts stored in model weights.","locator":"Abstract, fourth sentence","url":"https://arxiv.org/abs/2202.05262v1","publisher":"arXiv; Kevin Meng, David Bau, Alex Andonian and Yonatan Belinkov","event_date":"2022-02","available_at":"2022-02-10","explanation":"The authors used causal interventions to investigate factual recall and evaluated targeted factual edits on CounterFact. This supplies a concrete pre-post example of useful mechanistic understanding and repair without adding data or layers."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"204f8a6d91b7a4e2f1ab3067cf06e0a35ca013c6d6b0cb883ac8cd961bf2fe63","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T06:56:13Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-acced4-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-acced4-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-acced4-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T06:56:13Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-d1f821-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-d56a36","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-d56a36-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-d56a36#assessment-marcus-dear-elon-musk-here-are-five-things-d56a36-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-d56a36","subject_id":"gary-marcus","revision":1,"content_hash":"f9cf90f4d5f44e7c7e5488002fb96c4bdba4cf17f1433e1d7a9fbe6904c12945","normalized_proposition":"The long tail problem makes it easy to believe AI problems are much easier than they actually are.","kind":"causal","target":"the long tail problem","conditions":[],"original_horizon_text":null,"topic_tags":["reliability","capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-d56a36-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"The thing is, it’s easy to convince yourself that AI problems are much easier than they are actually are, because of the long tail problem.","locator":"paragraph 10","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-d56a36-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-d56a36","subject_id":"gary-marcus","normalized_proposition":"The long tail problem makes it easy to believe AI problems are much easier than they actually are.","kind":"causal","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"The thing is, it’s easy to convince yourself that AI problems are much easier than they are actually are, because of the long tail problem.","locator":"paragraph 10","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["easy to convince yourself","AI problems are much easier than they are actually are","because of the long tail problem"],"our_interpretation":["The ordinary reading is that success on frequently encountered cases can cause people to underestimate the difficulty of making AI reliable across uncommon cases.","Underestimating development difficulty is distinct from trusting an existing system too much.","The claim describes a general mechanism without a deadline or a specified population."],"support_condition":"Studies or documented causal analyses show that exposure dominated by successful common cases produces underestimation of AI problem difficulty, and connect that misjudgment to overlooked rare cases.","contradiction_condition":"Well-matched studies show that this exposure does not produce the proposed underestimation, or that another factor accounts for the effect while rare-case exposure has no material influence."},"axis":"causal_assessment","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Research on overtrust is consistent with part of Marcus's explanation: repeated successful interactions can encourage excessive confidence. But that does not establish that the long-tail problem causes people to underestimate AI engineering difficulty. The searches did not locate evidence that directly settles that causal claim.","limitations":["The closest empirical evidence uses a simulated robot task.","The speaker supplies neither a population nor a measure of perceived difficulty."],"evidence":[{"id":"e1","relation":"context","excerpt":"System failure is a rare occurrence.","locator":"General paradigm of overtrust, page 6","url":"https://www.mmi.ifi.lmu.de/pubdb/publications/pub/ullrich2021DevelopmentOfOvertrust/ullrich2021DevelopmentOfOvertrust.pdf","publisher":"Frontiers in Robotics and AI; institutional copy hosted by LMU Munich","event_date":"2021","available_at":"2021-04-13","explanation":"The study models successful interactions followed by uncommon failures and reports increasing trust during repeated successes. Its simulated pet-feeding task measures trust and checking behavior, not beliefs about how difficult AI engineering problems are."}],"what_would_resolve":"A study comparing common-case-only exposure with exposure including rare failures, measuring resulting estimates of AI development difficulty or remaining work against independently established performance limitations.","retrospective":true,"plain_reading":true},"payload_hash":"871b2d9884d1f58427c1cac370cd73ec3b37fb0ad201893a30765c42676580c8","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T06:56:13Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-acced4-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-acced4-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-acced4-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T06:56:13Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-d56a36-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-ed6b53","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-ed6b53-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-ed6b53#assessment-marcus-dear-elon-musk-here-are-five-things-ed6b53-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-ed6b53","subject_id":"gary-marcus","revision":1,"content_hash":"967264432a6b0ca36acbe5f2bf2fa4eb02af6b8c33dfbcba70f0cc11f9658c51","normalized_proposition":"Tesla will not ship level 5 self-driving cars in 2022 or 2023.","kind":"forecast","target":"Tesla","conditions":[],"original_horizon_text":"this year or next","topic_tags":["capabilities","reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-ed6b53-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"And it’s why I absolutely guarantee you won’t be shipping level 5 self-driving cars this year or next, no matter what you might have told Chris Anderson at TED.","locator":"paragraph 16","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-ed6b53-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-ed6b53","subject_id":"gary-marcus","normalized_proposition":"Tesla will not ship level 5 self-driving cars in 2022 or 2023.","kind":"forecast","conditions":[],"original_horizon_text":"this year or next","statement":{"excerpt":"And it’s why I absolutely guarantee you won’t be shipping level 5 self-driving cars this year or next, no matter what you might have told Chris Anderson at TED.","locator":"paragraph 16","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["absolutely guarantee","won’t be shipping","level 5 self-driving cars","this year or next"],"our_interpretation":["The addressee and context identify Tesla as the manufacturer.","This year or next means 2022 or 2023.","Shipping means supplying customers with functioning Level 5 capability, including activation through software updates.","Level 5 means full driving automation without human supervision across driver-manageable road conditions. Selling hardware intended for future autonomy does not qualify."],"support_condition":"Tesla's documented customer offerings remained below Level 5 throughout 2022 and 2023.","contradiction_condition":"Tesla supplied at least one customer with a vehicle offering functioning Level 5 automation during 2022 or 2023."},"axis":"forecast_outcome","outcome":"supported","horizon_state":"elapsed","evaluation_as_of":"2026-10-28","rationale":"Tesla's annual filings for both forecast years describe its available systems as driver assistance requiring a responsible human driver. These documented capabilities fall below Level 5. The deadline has elapsed, and the records support Marcus's prediction.","limitations":["The assessment concerns delivered capability, not whether Tesla sold an option named Full Self-Driving.","Annual disclosures establish the product offering rather than independently testing every vehicle.","This outcome does not establish Marcus's proposed explanation for the delay."],"evidence":[{"id":"e1","relation":"supports","excerpt":"advanced driver assist systems","locator":"Item 1, Self-Driving Development and Artificial Intelligence","url":"https://www.annualreports.com/HostedData/AnnualReportArchive/t/NASDAQ_TSLA_2022.pdf","publisher":"Tesla, Inc., mirrored by AnnualReports.com","event_date":"2022-12-31","available_at":"2023-01-31","explanation":"The annual filing describes Tesla's offerings as driver assistance and says the driver remains responsible for controlling the vehicle. That capability does not meet Level 5."},{"id":"e2","relation":"supports","excerpt":"advanced driver assist systems","locator":"Item 1, Self-Driving Development and Artificial Intelligence, page 5","url":"https://s203.q4cdn.com/655419027/files/doc_financials/2023/ar/tsla-20231231-gen.pdf","publisher":"Tesla, Inc.","event_date":"2023-12-31","available_at":"2024-01-29","explanation":"The following annual filing still describes the delivered systems as driver assistance, with the driver responsible for control. Together, the filings support non-delivery of Level 5 capability through the deadline."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"6bc5b8fabe289c22eddf037ac43e6d0347c8fe038c1052b9269ed52f9257853f","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T06:56:13Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-acced4-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-acced4-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-acced4-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"filing","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"filing","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T06:56:13Z","next_review_due_at":null,"horizon_now":"elapsed"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-ed6b53-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-f025f6","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-f025f6-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-f025f6#assessment-marcus-dear-elon-musk-here-are-five-things-f025f6-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-f025f6","subject_id":"gary-marcus","revision":1,"content_hash":"3f77b22fdbbd6cbc9d8049b8a41863f6862e09826baa1e21c38b830bcd743b12","normalized_proposition":"In 2029, AI will not be able to watch a movie and accurately describe what is going on.","kind":"forecast","target":"AI","conditions":[],"original_horizon_text":"In 2029","topic_tags":["capabilities","agi-timelines"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-f025f6-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"In 2029, AI will not be able to watch a movie and tell you accurately what is going on (what I called the comprehension challenge in The New Yorker, in 2014).","locator":"prediction 1 of 5","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-f025f6-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-f025f6","subject_id":"gary-marcus","normalized_proposition":"In 2029, AI will not be able to watch a movie and accurately describe what is going on.","kind":"forecast","conditions":[],"original_horizon_text":"In 2029","statement":{"excerpt":"In 2029, AI will not be able to watch a movie and tell you accurately what is going on (what I called the comprehension challenge in The New Yorker, in 2014).","locator":"prediction 1 of 5","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["In 2029","AI will not be able to watch a movie","tell you accurately what is going on","Who are the characters?","What are their conflicts and motivations?"],"our_interpretation":["Read this as a forecast about general movie comprehension, including events, characters, conflicts and inferred motivations.","A suitable test uses varied full movies and open-ended explanations, with controls against answering from memorized summaries. A hand-picked clip or isolated correct answer is insufficient.","Accuracy means ordinary competent comprehension, allowing reasonable differences in interpretation; it does not require perfect recall of every detail.","Use 2029-12-31 as the latest deadline. An earlier reproducible demonstration of the complete capability can refute the forecast under the early-resolution rule.","Assess this individual forecast independently of the surrounding wager about passing three challenges."],"support_condition":"Evidence covering the end of 2029 shows that leading available systems still cannot accurately explain full movies across a representative evaluation of events, characters, conflicts and motivations.","contradiction_condition":"At least one AI system demonstrates accurate full-movie comprehension across varied movies under a credible evaluation that tests the stated narrative elements."},"axis":"forecast_outcome","outcome":"unresolved","horizon_state":"not_due","evaluation_as_of":"2026-10-28","rationale":"The deadline is still ahead. The inspected movie-understanding research does not provide a decisive early demonstration of the complete task. Its reported shortcomings are evidence about tested systems, not proof that the forecast will hold in 2029.","limitations":["The search does not establish that no qualifying system exists.","There is no speaker-specified accuracy threshold.","Benchmark results cover selected models and films."],"evidence":[{"id":"e1","relation":"context","excerpt":"a substantial gap","locator":"Abstract, fourth sentence","url":"https://arxiv.org/abs/2506.06275v2","publisher":"arXiv; Emmanouil Zaranis and collaborators","event_date":"2026","available_at":"2026-10-19","explanation":"The study reports persistent difficulties in narrative understanding and improvements when relevant visual content is selected. It does not establish the complete capability required for early contradiction, and its results cannot settle performance in 2029."}],"what_would_resolve":"A reproducible full-movie evaluation meeting the early-resolution rule would contradict the forecast. Otherwise, evaluate available systems through 2029 using a disclosed rubric covering events, characters, conflicts and motivations.","retrospective":false,"plain_reading":true},"payload_hash":"ebe9e0119daf8367a5f1a9d3b11931bf53ef42252e546684809c81b984c09dcd","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T06:55:39Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-f025f6-2x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-f025f6-2x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-f025f6-2x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T06:55:39Z","next_review_due_at":null,"horizon_now":"not_due"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-f025f6-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-fcf450","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-fcf450-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-fcf450#assessment-marcus-dear-elon-musk-here-are-five-things-fcf450-a1"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-fcf450","subject_id":"gary-marcus","revision":1,"content_hash":"432bd40fbf5bbf86ce148c9296d666d6cdb4797605f74f23a3771ee3d00e3e06","normalized_proposition":"Deep learning lacks tools for verifiable code, and AGI by 2029 would probably require solving that soon.","kind":"conditional_forecast","target":"AGI by 2029","conditions":[{"text":"if AGI is to arrive by 2029","type":"antecedent","from_source":true}],"original_horizon_text":"soon","topic_tags":["agi-timelines","reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-fcf450-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"Things like programming verification might eventually help, but again, in deep learning we don’t yet have the tools for writing verifiable code. If you are to win the bet, we are probably going to have to solve that problem too, and soon.","locator":"paragraph 31","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-fcf450-a1","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-fcf450","subject_id":"gary-marcus","normalized_proposition":"Deep learning lacks tools for verifiable code, and AGI by 2029 would probably require solving that soon.","kind":"conditional_forecast","conditions":["if AGI is to arrive by 2029"],"original_horizon_text":"soon","statement":{"excerpt":"Things like programming verification might eventually help, but again, in deep learning we don’t yet have the tools for writing verifiable code. If you are to win the bet, we are probably going to have to solve that problem too, and soon.","locator":"paragraph 31","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["if AGI is to arrive by 2029","soon"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"1ccda1859a18749416a0ee80758ee6b3e95865ddfc9fb0f8ac71c43961962612","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:10:54Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:10:54Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-fcf450-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-ffdca9","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-ffdca9-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-dear-elon-musk-here-are-five-things-ffdca9#assessment-marcus-dear-elon-musk-here-are-five-things-ffdca9-a2"}],"claim":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-ffdca9","subject_id":"gary-marcus","revision":1,"content_hash":"8b842ed5a0924c6ee36960c80a17807aad851b929caacd44a946c43bc4eeaf23","normalized_proposition":"Outliers remain a major AI problem with no known robust solution.","kind":"observation","target":"outliers in AI","conditions":[],"original_horizon_text":null,"topic_tags":["reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-dear-elon-musk-here-are-five-things-ffdca9-o1","statement_date":"2022-05-31","statement_date_precision":"day","excerpt":"But you and I both know that they continue to be a major problem that as yet has no known robust solution.","locator":"paragraph 17","document":{"url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","title":"Dear Elon Musk, here are five things you might want to consider about AGI","publisher":"Marcus on AI","published_at":"2022-05-31T18:32:41Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-28T22:49:04Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"d64c073bb595b86ed0d5a52382668c601d5a8e7009353d00f0e54061bdd4f0f7","receipts":1}],"first_statement_date":"2022-05-31","related":[],"current":{"assessment_id":"marcus-dear-elon-musk-here-are-five-things-ffdca9-a2","payload":{"claim_id":"marcus-dear-elon-musk-here-are-five-things-ffdca9","subject_id":"gary-marcus","normalized_proposition":"Outliers remain a major AI problem with no known robust solution.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"But you and I both know that they continue to be a major problem that as yet has no known robust solution.","locator":"paragraph 17","url":"https://garymarcus.substack.com/p/dear-elon-musk-here-are-five-things","date":"2022-05-31","document_title":"Dear Elon Musk, here are five things you might want to consider about AGI","observed_at":"2026-10-28T22:49:04Z"},"criteria":{"from_speaker":["they continue to be a major problem","as yet","no known robust solution"],"our_interpretation":["They refers to outliers and unexpected circumstances discussed immediately before this sentence.","The ordinary reading concerns reliable handling of unfamiliar inputs and distribution shifts across applications. It does not deny effective methods for particular cases.","Because academic research has no fixed reporting quarter, the observation is assessed against the current 2022 research record through May 31. Earlier research supplies background.","A robust solution means broadly demonstrated reliability, not perfect performance on every mathematically possible input."],"support_condition":"Contemporaneous research documents substantial failures under unexpected inputs or distribution shifts, with available remedies remaining limited rather than constituting a broadly validated solution.","contradiction_condition":"Research available by the post establishes a known method that reliably resolves these failures across representative applications and unfamiliar conditions."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The February 16, 2022 survey identifies continuing obstacles to handling distribution shifts in graph data, including differences in graph structure, tasks and shift types [e1]. The February 21, 2022 experiments find that fine-tuning can worsen accuracy under distribution shift across ten tested datasets, while a two-stage training method improves those results [e2]. These findings support the observation, as of May 31, 2022, that unfamiliar inputs remained a substantial unresolved problem: they document obstacles and useful remedies without establishing a broadly reliable solution [e1; e2].","limitations":["The absence of a broadly validated solution is an inference from the inspected public research record, not proof that every possible solution was absent.","The graph survey and selected distribution-shift datasets do not cover every application or kind of outlier.","The evidence does not establish that outlier problems are inherently unsolvable or validate every surrounding criticism of deep learning."],"evidence":[{"id":"e1","relation":"supports","excerpt":"highly non-trivial","locator":"Section 1, Introduction","url":"https://arxiv.org/pdf/2202.07987v1","publisher":"arXiv; Haoyang Li and colleagues","event_date":"2022-02-16","available_at":"2022-02-16","explanation":"The survey identifies unresolved obstacles involving graph structure, differing tasks, and different kinds of distribution shift. It also describes progress, supporting the distinction between useful remedies and a general solution."},{"id":"e2","relation":"supports","excerpt":"tradeoff","locator":"Abstract","url":"https://arxiv.org/abs/2202.10054v1","publisher":"arXiv; Ananya Kumar and colleagues","event_date":"2022-02-21","available_at":"2022-02-21","explanation":"Experiments across ten distribution-shift datasets show that improving ordinary accuracy can reduce accuracy after a shift. The proposed two-stage training method improves results, but the paper does not establish a general solution to unexpected inputs."},{"id":"e3","relation":"context","excerpt":"open challenges","locator":"Abstract, final sentence","url":"https://arxiv.org/abs/2110.11334v1","publisher":"arXiv; Jingkang Yang and colleagues","event_date":"2021-10-21","available_at":"2021-10-21","explanation":"This earlier survey covers anomaly, novelty, open-set, outlier, and out-of-distribution detection. It supplies background on the breadth of the problem and the existence of many partial approaches."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"e9434f4c415086f41afe30f709cb72a7e300304f2202dbbc1a2e9719ad7dccb5","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T07:09:09Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-f025f6-2x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-dear-elon-musk-here-are-five-things-f025f6-2x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-dear-elon-musk-here-are-five-things-31b9db-2x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e3","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T07:09:09Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-dear-elon-musk-here-are-five-things-ffdca9-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-1e0fda","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-horse-rides-astronaut-1e0fda-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-1e0fda#assessment-marcus-horse-rides-astronaut-1e0fda-a1"}],"claim":{"claim_id":"marcus-horse-rides-astronaut-1e0fda","subject_id":"gary-marcus","revision":2,"content_hash":"280279d00436a50549932e8a3c19eef6f0b40d02fb248546544c7fbb6f9bb1fe","normalized_proposition":"From the few clues that have filtered out, there is enormous reason to be skeptical of Google's claim that Imagen has a deep level of language understanding, though without access to the system skeptics can only make guesses.","kind":"normative","target":"Imagen","conditions":[{"text":"from the few clues that have filtered out","type":"qualifier","from_source":true},{"text":"Google won't allow skeptics to try the system, so we can only make guesses","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-horse-rides-astronaut-1e0fda-o1","statement_date":"2022-05-28","statement_date_precision":"day","excerpt":"Against a long history of overclaiming in AI, such strong statements demand an awful lot of evidence. Because Google won’t allow skeptics to try the system, we can only make guesses. But from the few clues that have filtered out, there is enormous reason for skepticism.","locator":"paragraph 31","document":{"url":"https://garymarcus.substack.com/p/horse-rides-astronaut","title":"Horse rides astronaut","publisher":"Marcus on AI","published_at":"2022-05-28T14:29:44Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:21Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"e4a8233fec988ccb75c5e8a45ac1ad325ab13313a43cd661957d6e57ebb49d48","receipts":1}],"first_statement_date":"2022-05-28","related":[],"current":{"assessment_id":"marcus-horse-rides-astronaut-1e0fda-a1","payload":{"claim_id":"marcus-horse-rides-astronaut-1e0fda","subject_id":"gary-marcus","normalized_proposition":"From the few clues that have filtered out, there is enormous reason to be skeptical of Google's claim that Imagen has a deep level of language understanding, though without access to the system skeptics can only make guesses.","kind":"normative","conditions":["from the few clues that have filtered out","Google won't allow skeptics to try the system, so we can only make guesses"],"original_horizon_text":null,"statement":{"excerpt":"Against a long history of overclaiming in AI, such strong statements demand an awful lot of evidence. Because Google won’t allow skeptics to try the system, we can only make guesses. But from the few clues that have filtered out, there is enormous reason for skepticism.","locator":"paragraph 31","url":"https://garymarcus.substack.com/p/horse-rides-astronaut","date":"2022-05-28","document_title":"Horse rides astronaut","observed_at":"2026-10-29T00:05:21Z"},"criteria":{"from_speaker":["from the few clues that have filtered out","Google won't allow skeptics to try the system, so we can only make guesses"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"102741a784a3d69bd6daee21f92b0382627a56cac5c5fdbc6ecf5d4a4cd19307","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-horse-rides-astronaut-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-horse-rides-astronaut-1e0fda-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-21930b","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-horse-rides-astronaut-21930b-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-21930b#assessment-marcus-horse-rides-astronaut-21930b-a1"}],"claim":{"claim_id":"marcus-horse-rides-astronaut-21930b","subject_id":"gary-marcus","revision":2,"content_hash":"769d1bd2a4e67f11b26c7a5bdad290a80d97a8dacc50a7366b0ac558c65a6d94","normalized_proposition":"Imagen's failure on 'a horse riding an astronaut' casts doubt on its success with 'an astronaut riding a horse' and suggests that this success comes not from extracting word meanings and combining them by syntax, but from something more holistic and approximate, a bit like keyword matching.","kind":"causal","target":"Imagen","conditions":[{"text":"casts doubt / suggests","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-horse-rides-astronaut-21930b-o1","statement_date":"2022-05-28","statement_date_precision":"day","excerpt":"it casts doubt on “an astronaut riding a horse”, and suggests success there is not because the network knows how to extract meanings from individual words","locator":"paragraph 91","document":{"url":"https://garymarcus.substack.com/p/horse-rides-astronaut","title":"Horse rides astronaut","publisher":"Marcus on AI","published_at":"2022-05-28T14:29:44Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:21Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"e4a8233fec988ccb75c5e8a45ac1ad325ab13313a43cd661957d6e57ebb49d48","receipts":1}],"first_statement_date":"2022-05-28","related":[],"current":{"assessment_id":"marcus-horse-rides-astronaut-21930b-a1","payload":{"claim_id":"marcus-horse-rides-astronaut-21930b","subject_id":"gary-marcus","normalized_proposition":"Imagen's failure on 'a horse riding an astronaut' casts doubt on its success with 'an astronaut riding a horse' and suggests that this success comes not from extracting word meanings and combining them by syntax, but from something more holistic and approximate, a bit like keyword matching.","kind":"causal","conditions":["casts doubt / suggests"],"original_horizon_text":null,"statement":{"excerpt":"it casts doubt on “an astronaut riding a horse”, and suggests success there is not because the network knows how to extract meanings from individual words","locator":"paragraph 91","url":"https://garymarcus.substack.com/p/horse-rides-astronaut","date":"2022-05-28","document_title":"Horse rides astronaut","observed_at":"2026-10-29T00:05:21Z"},"criteria":{"from_speaker":["casts doubt / suggests"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"f8330d842eab0545918b2dd76e094b225c2890e6e67bbd4178d27e2a33238524","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-horse-rides-astronaut-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-horse-rides-astronaut-21930b-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-4f1bbd","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-horse-rides-astronaut-4f1bbd-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-4f1bbd#assessment-marcus-horse-rides-astronaut-4f1bbd-a1"}],"claim":{"claim_id":"marcus-horse-rides-astronaut-4f1bbd","subject_id":"gary-marcus","revision":1,"content_hash":"2b29b617b3d214832bc4d7b1c6ce03c0ef9d227b91d2e1dfae32061760104f91","normalized_proposition":"Imagen lacks deep understanding of language despite Google's PR.","kind":"observation","target":"Imagen","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-horse-rides-astronaut-4f1bbd-o1","statement_date":"2022-05-28","statement_date_precision":"day","excerpt":"That’s where the deep understanding of language should live—but, despite the PR, it isn’t there.","locator":"paragraph 83","document":{"url":"https://garymarcus.substack.com/p/horse-rides-astronaut","title":"Horse rides astronaut","publisher":"Marcus on AI","published_at":"2022-05-28T14:29:44Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:21Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"e4a8233fec988ccb75c5e8a45ac1ad325ab13313a43cd661957d6e57ebb49d48","receipts":1}],"first_statement_date":"2022-05-28","related":[],"current":{"assessment_id":"marcus-horse-rides-astronaut-4f1bbd-a1","payload":{"claim_id":"marcus-horse-rides-astronaut-4f1bbd","subject_id":"gary-marcus","normalized_proposition":"Imagen lacks deep understanding of language despite Google's PR.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"That’s where the deep understanding of language should live—but, despite the PR, it isn’t there.","locator":"paragraph 83","url":"https://garymarcus.substack.com/p/horse-rides-astronaut","date":"2022-05-28","document_title":"Horse rides astronaut","observed_at":"2026-10-29T00:05:21Z"},"criteria":{"from_speaker":["deep understanding of language","despite the PR","it isn’t there"],"our_interpretation":["In this context, the ordinary reading is reliable understanding of who does what to whom in compositional image instructions.","This is a behavioral claim about Imagen as a complete system, not a claim about consciousness or the absence of every language capability.","I use the original model's public evaluation window, from its May 23 announcement through the May 28 post, because the sentence describes an ongoing capability without specifying a reporting period.","Neither occasional mistakes nor occasional successes alone settle the claim; evidence should show a recurring weakness or robust success on the relevant instruction types."],"support_condition":"Evaluations of the original Imagen show recurring failures to preserve basic roles or relationships in compositional instructions, inconsistent with reliable understanding in the sense discussed.","contradiction_condition":"Controlled evaluations show reliable handling of the relevant role reversals, relationships, and equivalent paraphrases, with the cited failures shown to be isolated exceptions."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Under the contextual reading of reliable compositional instruction understanding, the original paper supports Marcus: its authors acknowledge alignment difficulties in the category containing the disputed prompt. Imagen's strong comparative results are material qualifications, but outperforming other models does not demonstrate reliable handling of these instructions. This finding does not establish that Imagen had no language capabilities or identify the internal cause of its failures.","limitations":["The result depends on interpreting deep understanding through the role-and-relationship behavior discussed in the post.","The evidence is the authors' qualitative category assessment, not a comprehensive paraphrase robustness study.","It applies to the original Imagen, not subsequent versions."],"evidence":[{"id":"e1","relation":"supports","excerpt":"struggle generating\nwell aligned images","locator":"Figure A.19 caption, page 33, Conflicting category","url":"https://www.3dvar.com/Saharia2022Photorealistic.pdf","publisher":"Google Research, Brain Team; copy hosted by 3DVAR","event_date":"2022-05-23","available_at":"2022-05-23","explanation":"The original authors identify a category-level alignment weakness in the figure containing the horse-riding-an-astronaut prompt. This supports the contextual reading that reliable compositional instruction understanding was missing."},{"id":"e2","relation":"context","excerpt":"Human raters strongly prefer Imagen","locator":"DrawBench: new comprehensive challenging benchmark","url":"https://imagen.research.google/","publisher":"Google Research","event_date":"2022-05-23","available_at":"2022-05-23","explanation":"The favorable comparative evaluation cautions against interpreting Marcus as showing no useful language capability. Preference over competing systems does not establish reliable understanding of the specific relationships at issue."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"6ef86bd16286ef70c1a77403b95013abf137e2bf02f31ee1d37664337d64a029","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T07:30:34Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-20af65-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-horse-rides-astronaut-20af65-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-20af65-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T07:30:34Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-horse-rides-astronaut-4f1bbd-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-61347d","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-horse-rides-astronaut-61347d-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-61347d#assessment-marcus-horse-rides-astronaut-61347d-a1"}],"claim":{"claim_id":"marcus-horse-rides-astronaut-61347d","subject_id":"gary-marcus","revision":1,"content_hash":"554434e21db75dd44089593f4ae422884b2c1ffd369df53f8daf4f9f69a6db33","normalized_proposition":"DALL-E 2 is easily fooled, especially by complex sentences and words requiring deep language understanding.","kind":"observation","target":"DALL-E 2","conditions":[{"text":"especially complex sentences and words like 'similar'","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities","reliability"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-horse-rides-astronaut-61347d-o1","statement_date":"2022-05-28","statement_date_precision":"day","excerpt":"As Ernie Davis, Scott Aaronson, and I showed, a week or so after the release, it’s easily fooled, especially with complex sentences and words like “similar” that would require a deep understanding of language to be interpreted correctly","locator":"paragraph 18","document":{"url":"https://garymarcus.substack.com/p/horse-rides-astronaut","title":"Horse rides astronaut","publisher":"Marcus on AI","published_at":"2022-05-28T14:29:44Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:21Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"e4a8233fec988ccb75c5e8a45ac1ad325ab13313a43cd661957d6e57ebb49d48","receipts":1}],"first_statement_date":"2022-05-28","related":[],"current":{"assessment_id":"marcus-horse-rides-astronaut-61347d-a1","payload":{"claim_id":"marcus-horse-rides-astronaut-61347d","subject_id":"gary-marcus","normalized_proposition":"DALL-E 2 is easily fooled, especially by complex sentences and words requiring deep language understanding.","kind":"observation","conditions":["especially complex sentences and words like 'similar'"],"original_horizon_text":null,"statement":{"excerpt":"As Ernie Davis, Scott Aaronson, and I showed, a week or so after the release, it’s easily fooled, especially with complex sentences and words like “similar” that would require a deep understanding of language to be interpreted correctly","locator":"paragraph 18","url":"https://garymarcus.substack.com/p/horse-rides-astronaut","date":"2022-05-28","document_title":"Horse rides astronaut","observed_at":"2026-10-29T00:05:21Z"},"criteria":{"from_speaker":["a week or so after the release","it’s easily fooled","especially with complex sentences and words like “similar”"],"our_interpretation":["The context explicitly places these observations in April 2022. The target is the original DALL-E 2.","The plain reading is that complex captions repeatedly produce substantial errors in objects, attributes or relationships, with particular difficulty interpreting relational language.","“Similar” is an illustrative example. No numerical failure rate or precise meaning of “deep understanding” is supplied.","Correctness concerns whether images satisfy the caption, rather than their visual quality."],"support_condition":"Independent records of the April 2022 model show repeated substantive failures on complex relational captions, including evidence that linguistic complexity or relational expressions create particular difficulty.","contradiction_condition":"Comparable controlled tests of the April 2022 model show reliable interpretation of complex relational captions, without the claimed particular difficulty."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The original model paper documents attribute-binding problems, so there is evidence for a narrower weakness. The retrieved independent evidence does not settle the stronger claim about frequent failures, especially on complex sentences and relational expressions, during April 2022. Marcus’s own study is excluded by the evidence rules. Later evaluations cannot automatically establish the earlier state.","limitations":["Unresolved does not mean the claim was false.","The evidence supports some compositional weaknesses without establishing the full comparison Marcus makes."],"evidence":[{"id":"e1","relation":"context","excerpt":"worse at binding attributes to objects","locator":"Section 7, Limitations and Risks","url":"https://ar5iv.labs.arxiv.org/html/2204.06125","publisher":"OpenAI authors, published through arXiv","event_date":"2022-04","available_at":"2022-04-13","explanation":"The original model paper documents difficulty assigning properties to objects. This supports a narrower weakness, but does not establish frequent failures specifically associated with complex syntax or expressions such as “similar.”"}],"what_would_resolve":"Independent April prompt-output records, or a documented evaluation of the preserved April model, testing complex relational captions against simpler controls with all outputs retained.","retrospective":true,"plain_reading":true},"payload_hash":"04dc2958e8a30ee187052d2a03b4f3786da4f17f5c3b23d8bb5ed75715a95832","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T07:30:34Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-20af65-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-horse-rides-astronaut-20af65-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-20af65-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T07:30:34Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-horse-rides-astronaut-61347d-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-63b833","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-horse-rides-astronaut-63b833-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-63b833#assessment-marcus-horse-rides-astronaut-63b833-a1"}],"claim":{"claim_id":"marcus-horse-rides-astronaut-63b833","subject_id":"gary-marcus","revision":2,"content_hash":"b2129fdb8fff457d2cae48e575ebb3cc9682c87fddd6593070a7ed8ac7cd52d0","normalized_proposition":"Since Imagen can draw horses riding astronauts when the instructions are specific enough, its failure to draw one from the prompt 'a horse riding an astronaut' can only be explained by the system not understanding what that prompt means.","kind":"causal","target":"Imagen","conditions":[{"text":"since Imagen can draw horses riding astronauts if the instructions are specific enough","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-horse-rides-astronaut-63b833-o1","statement_date":"2022-05-28","statement_date_precision":"day","excerpt":"Since we know that Imagen can draw images of horses riding astronauts (if the instructions are specific enough) we know that the failure to draw a horse riding an astronaut given the prompt “a horse riding an astronaut” can only have one explanation: the system doesn’t understand what the prompt “a horse riding an astronaut” means.","locator":"paragraph 90","document":{"url":"https://garymarcus.substack.com/p/horse-rides-astronaut","title":"Horse rides astronaut","publisher":"Marcus on AI","published_at":"2022-05-28T14:29:44Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:21Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"e4a8233fec988ccb75c5e8a45ac1ad325ab13313a43cd661957d6e57ebb49d48","receipts":1}],"first_statement_date":"2022-05-28","related":[],"current":{"assessment_id":"marcus-horse-rides-astronaut-63b833-a1","payload":{"claim_id":"marcus-horse-rides-astronaut-63b833","subject_id":"gary-marcus","normalized_proposition":"Since Imagen can draw horses riding astronauts when the instructions are specific enough, its failure to draw one from the prompt 'a horse riding an astronaut' can only be explained by the system not understanding what that prompt means.","kind":"causal","conditions":["since Imagen can draw horses riding astronauts if the instructions are specific enough"],"original_horizon_text":null,"statement":{"excerpt":"Since we know that Imagen can draw images of horses riding astronauts (if the instructions are specific enough) we know that the failure to draw a horse riding an astronaut given the prompt “a horse riding an astronaut” can only have one explanation: the system doesn’t understand what the prompt “a horse riding an astronaut” means.","locator":"paragraph 90","url":"https://garymarcus.substack.com/p/horse-rides-astronaut","date":"2022-05-28","document_title":"Horse rides astronaut","observed_at":"2026-10-29T00:05:21Z"},"criteria":{"from_speaker":["Imagen can draw images of horses riding astronauts","if the instructions are specific enough","can only have one explanation","the system doesn’t understand what the prompt","a horse riding an astronaut","means"],"our_interpretation":["This concerns the original Imagen's May 2022 prompt failure.","The success of more specific instructions is an asserted premise, not a future conditional forecast.","Only one explanation commits to causal exclusivity. Evidence that some prompt changes help is insufficient unless competing causes of the original failure are excluded.","The claim does not specifically identify a neural-network component, so understanding is assessed through preservation and use of the requested semantic roles."],"support_condition":"Tests of the original Imagen verify the premise and establish failed semantic interpretation as the explanation of the original failure while excluding downstream conditioning, generation, sampling, and selection explanations.","contradiction_condition":"Tests of the original Imagen establish another cause of the failure, including cases where the requested roles are represented correctly but the generation process fails to express them."},"axis":"causal_assessment","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The premise is supported by Neyshabur's report that more specific wording improves the chance of success. The exclusive explanation is not established. Successful rewording does not by itself show where the original attempt failed. Research on other generators cannot supply the missing diagnosis for Imagen, and merely proposing alternatives would not justify calling Marcus contradicted.","limitations":["The available demonstration is not a controlled causal experiment.","An output-level failure cannot by itself distinguish absent semantic interpretation from failure to use an adequate representation.","The post's acknowledged uncertainty limits confidence but does not erase its explicit exclusivity claim."],"evidence":[{"id":"e1","relation":"supports","excerpt":"helps increase the chance","locator":"Neyshabur's second sentence","url":"https://twitter.com/bneyshabur/status/1529506107655303168","publisher":"Behnam Neyshabur","event_date":"2022-05-25","available_at":"2022-05-25","explanation":"The firsthand report supports the premise that more specific instructions can produce the intended scene. It supplies no test establishing a unique cause for failures with the original wording."},{"id":"e2","relation":"context","excerpt":"stablediffusion 2.1 text-to-image model.","locator":"Abstract, third sentence","url":"https://arxiv.org/abs/2306.01755","publisher":"Charles Lovering and Ellie Pavlick, arXiv","event_date":"2023","available_at":"2023-10-25","explanation":"This study relates training frequencies to success on reversed relationships. Because it tests another model, it cannot establish an alternative cause of the original Imagen failure."}],"what_would_resolve":"A controlled investigation of the original Imagen that identifies the cause of the disputed failures and either excludes or demonstrates competing explanations.","retrospective":true,"plain_reading":true},"payload_hash":"faa6c48fd3b9ed6c0520d23624de08e8df83cc4ddd2d1f3de7043d49d793216a","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T07:30:19Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-63b833-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-horse-rides-astronaut-63b833-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-63b833-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"demonstration","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T07:30:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-horse-rides-astronaut-63b833-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-6501e4","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-horse-rides-astronaut-6501e4-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-6501e4#assessment-marcus-horse-rides-astronaut-6501e4-a2"}],"claim":{"claim_id":"marcus-horse-rides-astronaut-6501e4","subject_id":"gary-marcus","revision":1,"content_hash":"8c2b7ba16d693fc0c40a69fcc68899d4271d73b191962aeac77947b8f71712f7","normalized_proposition":"Imagen's failure on 'a horse riding an astronaut' was not due to common sense.","kind":"causal","target":"Imagen","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-horse-rides-astronaut-6501e4-o1","statement_date":"2022-05-28","statement_date_precision":"day","excerpt":"And, no, it wasn’t about common sense at all.","locator":"paragraph 65","document":{"url":"https://garymarcus.substack.com/p/horse-rides-astronaut","title":"Horse rides astronaut","publisher":"Marcus on AI","published_at":"2022-05-28T14:29:44Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:21Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"e4a8233fec988ccb75c5e8a45ac1ad325ab13313a43cd661957d6e57ebb49d48","receipts":1}],"first_statement_date":"2022-05-28","related":[],"current":{"assessment_id":"marcus-horse-rides-astronaut-6501e4-a2","payload":{"claim_id":"marcus-horse-rides-astronaut-6501e4","subject_id":"gary-marcus","normalized_proposition":"Imagen's failure on 'a horse riding an astronaut' was not due to common sense.","kind":"causal","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"And, no, it wasn’t about common sense at all.","locator":"paragraph 65","url":"https://garymarcus.substack.com/p/horse-rides-astronaut","date":"2022-05-28","document_title":"Horse rides astronaut","observed_at":"2026-10-29T00:05:21Z"},"criteria":{"from_speaker":["it wasn’t about common sense at all","All good guesses, all wrong"],"our_interpretation":["The claim concerns the cause of original Imagen’s May 2022 failures on the horse-riding-astronaut prompt.","In context, “common sense” means a plausibility judgment or learned expectation that makes the model favor ordinary riding arrangements.","Successful rewording can rule out an absolute inability or unconditional refusal. It does not by itself rule out a contributing plausibility preference.","The categorical phrase “at all” requires stronger evidence than showing that wording affects the result."],"support_condition":"Controlled comparisons or model interventions show that plausibility-based judgment did not cause the original failure, while identifying a sufficient alternative mechanism.","contradiction_condition":"Controlled evidence shows that a plausibility-based judgment or expectation causally contributed to the role reversal."},"axis":"causal_assessment","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"On May 25, 2022, Neyshabur reported that adding “on back of” or “on shoulders of” improved Imagen’s chance of depicting a horse riding an astronaut [e5]. The demonstration does not separate wording effects from learned plausibility preferences, so it cannot establish whether those preferences contributed to the original failure [e5]. The claim that common sense played no causal role remains unresolved [e5].","limitations":["The demonstration provides no repeated-sample success rates or controlled comparison separating wording from scene plausibility [e5].","Successful rewording establishes neither the presence nor the absence of a plausibility-based contribution to the original failure [e5]."],"evidence":[{"id":"e5","relation":"context","excerpt":"helps increase the chance of getting it right!","locator":"Neyshabur’s post, second sentence","url":"https://twitter.com/bneyshabur/status/1529506107655303168","publisher":"Behnam Neyshabur","event_date":"2022-05-25","available_at":"2022-05-25","explanation":"The researcher reports improved success after specifying the riding position. This challenges an unconditional refusal explanation, but does not isolate or exclude a contributing preference for plausible scenes."}],"what_would_resolve":"Controlled tests on original Imagen, using repeated samples and matched prompt variants, that separate wording from plausibility effects. These would need to establish a causal contribution from plausibility preferences, or exclude that contribution while identifying a sufficient alternative mechanism.","retrospective":true,"plain_reading":true},"payload_hash":"534d8c7db1298d6795bd9bf2718f9ea04fc90c01fc0a316812a786c13be3c56a","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T07:41:18Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-63b833-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-horse-rides-astronaut-63b833-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-6501e4-2x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e5","source_type":"demonstration","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T07:41:18Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-horse-rides-astronaut-6501e4-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-792402","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-horse-rides-astronaut-792402-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-792402#assessment-marcus-horse-rides-astronaut-792402-a1"}],"claim":{"claim_id":"marcus-horse-rides-astronaut-792402","subject_id":"gary-marcus","revision":1,"content_hash":"165dd46574db8584124f3850efbb6e152d0babd6eb41dcacabea2c762268cd8d","normalized_proposition":"Imagen almost certainly does not have a deep understanding of language.","kind":"observation","target":"Imagen","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-horse-rides-astronaut-792402-o1","statement_date":"2022-05-28","statement_date_precision":"day","excerpt":"The answer to the second question—does Imagen have a deep understanding of language—is almost certainly “no”: if a system can draw a given picture with incantation X, but not with perfectly sensible alternative incantation Y, it’s pretty hard to say that the system understands Y.","locator":"paragraph 89","document":{"url":"https://garymarcus.substack.com/p/horse-rides-astronaut","title":"Horse rides astronaut","publisher":"Marcus on AI","published_at":"2022-05-28T14:29:44Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:21Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"e4a8233fec988ccb75c5e8a45ac1ad325ab13313a43cd661957d6e57ebb49d48","receipts":1}],"first_statement_date":"2022-05-28","related":[],"current":{"assessment_id":"marcus-horse-rides-astronaut-792402-a1","payload":{"claim_id":"marcus-horse-rides-astronaut-792402","subject_id":"gary-marcus","normalized_proposition":"Imagen almost certainly does not have a deep understanding of language.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"The answer to the second question—does Imagen have a deep understanding of language—is almost certainly “no”: if a system can draw a given picture with incantation X, but not with perfectly sensible alternative incantation Y, it’s pretty hard to say that the system understands Y.","locator":"paragraph 89","url":"https://garymarcus.substack.com/p/horse-rides-astronaut","date":"2022-05-28","document_title":"Horse rides astronaut","observed_at":"2026-10-29T00:05:21Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found that evidence could not settle it: one found it too vague to test: informed readers would test it in different ways and reach different answers; the other found it stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading","probability_language"]},"payload_hash":"fc32f20150d5cffcb93e14d792c8bf7f080c91da00e7117200738d2dc0d03c37","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:10:54Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-horse-rides-astronaut-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:10:54Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-horse-rides-astronaut-792402-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-889e7b","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-horse-rides-astronaut-889e7b-a3","status":"current","url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-889e7b#assessment-marcus-horse-rides-astronaut-889e7b-a3"}],"claim":{"claim_id":"marcus-horse-rides-astronaut-889e7b","subject_id":"gary-marcus","revision":1,"content_hash":"53b3aeae627477791fe578c671e8eb8f0841e44de21b9b64cc309aafb6720754","normalized_proposition":"Imagen's failure lies in the connection between language and images, not in image generation.","kind":"causal","target":"Imagen","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-horse-rides-astronaut-889e7b-o1","statement_date":"2022-05-28","statement_date_precision":"day","excerpt":"The problem was not on the image generation side, it was in the connection between language and images.","locator":"paragraph 82","document":{"url":"https://garymarcus.substack.com/p/horse-rides-astronaut","title":"Horse rides astronaut","publisher":"Marcus on AI","published_at":"2022-05-28T14:29:44Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:21Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"e4a8233fec988ccb75c5e8a45ac1ad325ab13313a43cd661957d6e57ebb49d48","receipts":1}],"first_statement_date":"2022-05-28","related":[],"current":{"assessment_id":"marcus-horse-rides-astronaut-889e7b-a3","payload":{"claim_id":"marcus-horse-rides-astronaut-889e7b","subject_id":"gary-marcus","normalized_proposition":"Imagen's failure lies in the connection between language and images, not in image generation.","kind":"causal","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"The problem was not on the image generation side, it was in the connection between language and images.","locator":"paragraph 82","url":"https://garymarcus.substack.com/p/horse-rides-astronaut","date":"2022-05-28","document_title":"Horse rides astronaut","observed_at":"2026-10-29T00:05:21Z"},"criteria":{"from_speaker":["not on the image generation side","in the connection between language and images"],"our_interpretation":["The claim concerns the original Imagen and the May 2022 horse-and-astronaut example.","I read the distinction as a causal claim about translating the instruction into an image versus generating the scene. It does not specifically name the text encoder.","Success with another prompt establishes some capacity to depict the scene, but does not by itself identify the cause of failure with the original prompt."],"support_condition":"Comparisons on the original Imagen establish that it can depict the requested scene and locate the original failure in language-to-image mapping, excluding an image-generation limitation as the explanation.","contradiction_condition":"Controlled evidence on the original Imagen shows that the prompt's meaning was adequately conveyed and that an image-generation limitation caused the failure."},"axis":"causal_assessment","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"On May 25, 2022, Neyshabur reported that Imagen could depict a horse riding an astronaut after a small prompt change [e1]. This demonstrates some capacity to produce the scene, but does not establish why the original prompt failed [e1]. The demonstration neither locates that failure in the connection between language and images nor excludes an image-generation limitation [e1]. The claimed cause remains unresolved [e1].","limitations":["The demonstration reports no sample counts, success rates, seeds, or selection procedures [e1].","It provides no controlled comparison that separates language-to-image mapping failures from image-generation failures [e1]."],"evidence":[{"id":"e1","relation":"context","excerpt":"minimal change","locator":"Neyshabur's opening sentence","url":"https://twitter.com/bneyshabur/status/1529506103708602369","publisher":"Behnam Neyshabur","event_date":"2022-05-25","available_at":"2022-05-25","explanation":"This supports the narrower premise that changing the prompt can produce the intended scene. It does not isolate which process caused the original failure."}],"what_would_resolve":"Controlled tests on the original Imagen using matched prompts, repeated samples, and disclosed selection procedures. Measurements or interventions must establish whether language-to-image mapping caused the original failure or whether generation failed despite adequate conveyance of the prompt's meaning.","retrospective":true,"plain_reading":true},"payload_hash":"f69f6ddfd4ca6bccaf980273f89f1260b000ecafdf9d3ff1ac10f5608be9e279","policy_version":"2026-10-28.7","recorded_at":"2026-10-31T19:22:04Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-63b833-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-horse-rides-astronaut-63b833-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-889e7b-1x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"demonstration","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-31T19:22:04Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-horse-rides-astronaut-889e7b-assessment-published-a3","marcus-horse-rides-astronaut-889e7b-held-a2"],"last_substantive_change_at":"2026-10-31T22:29:51Z"}},{"url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-a6cd19","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-horse-rides-astronaut-a6cd19-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-a6cd19#assessment-marcus-horse-rides-astronaut-a6cd19-a1"}],"claim":{"claim_id":"marcus-horse-rides-astronaut-a6cd19","subject_id":"gary-marcus","revision":1,"content_hash":"ba73b1959d8fc9a8df29d73365ab37d1726d5c9370254932e0b0fa6806aed6ae","normalized_proposition":"Imagen has no understanding of weight, mass or density.","kind":"observation","target":"Imagen","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-horse-rides-astronaut-a6cd19-o1","statement_date":"2022-05-28","statement_date_precision":"day","excerpt":"(In reality, Imagen has no idea what weight, mass, or density are.)","locator":"paragraph 63","document":{"url":"https://garymarcus.substack.com/p/horse-rides-astronaut","title":"Horse rides astronaut","publisher":"Marcus on AI","published_at":"2022-05-28T14:29:44Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:21Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"e4a8233fec988ccb75c5e8a45ac1ad325ab13313a43cd661957d6e57ebb49d48","receipts":1}],"first_statement_date":"2022-05-28","related":[],"current":{"assessment_id":"marcus-horse-rides-astronaut-a6cd19-a1","payload":{"claim_id":"marcus-horse-rides-astronaut-a6cd19","subject_id":"gary-marcus","normalized_proposition":"Imagen has no understanding of weight, mass or density.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"(In reality, Imagen has no idea what weight, mass, or density are.)","locator":"paragraph 63","url":"https://garymarcus.substack.com/p/horse-rides-astronaut","date":"2022-05-28","document_title":"Horse rides astronaut","observed_at":"2026-10-29T00:05:21Z"},"criteria":{"from_speaker":["Imagen has no idea","weight, mass, or density"],"our_interpretation":["The target is the original Imagen system discussed in May 2022, not later Imagen versions.","For this ongoing state, the target period is May 2022 through publication, the period of the model’s initial public report.","The ordinary behavioral reading is that Imagen lacks systematic usable competence with these physical concepts.","This does not require deciding whether a machine has subjective understanding. Neither a single implausible image nor the architecture alone establishes absence of competence."],"support_condition":"Controlled evaluations of the original model consistently fail to demonstrate usable knowledge of weight, mass and density across varied prompts, after accounting for general image-generation and language-following errors.","contradiction_condition":"Controlled evaluations demonstrate systematic, transferable competence with one or more of these concepts, contradicting the blanket assertion of no understanding."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The searched records do not settle whether original Imagen had usable competence with weight, mass or density. General image benchmarks and failures on a horse–astronaut prompt do not establish the claimed absence of all such competence. No controlled evaluation of these concepts in the original model was located.","limitations":["The behavioral interpretation makes the claim testable but does not settle philosophical questions about understanding.","Missing evaluations are not evidence that the model lacked the concepts."],"evidence":[{"id":"e4","relation":"context","excerpt":"a comprehensive and challenging benchmark","locator":"Abstract, DrawBench introduction","url":"https://arxiv.org/abs/2205.11487","publisher":"Google Research authors, published through arXiv","event_date":"2022-05","available_at":"2022-05-23","explanation":"The original report describes a text-to-image evaluation framework. The inspected evaluation does not establish either systematic competence or its absence for weight, mass and density."}],"what_would_resolve":"Controlled tests of the preserved original model across weight, mass and density relationships, with matched language and rendering controls and independently scored outputs.","retrospective":true,"plain_reading":true},"payload_hash":"11fdc15069d4624f75994d922af3cf57712a8dc2c132ad50df6d31390bce906e","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T07:30:19Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-63b833-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-horse-rides-astronaut-63b833-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-63b833-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e4","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T07:30:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-horse-rides-astronaut-a6cd19-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-c336ee","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-horse-rides-astronaut-c336ee-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-c336ee#assessment-marcus-horse-rides-astronaut-c336ee-a1"}],"claim":{"claim_id":"marcus-horse-rides-astronaut-c336ee","subject_id":"gary-marcus","revision":1,"content_hash":"e654bb90172bad78e916f8c0d2568b85b1337ebe54320138c152ea851b469ff8","normalized_proposition":"DALL-E 2 is not as smart as it appears.","kind":"observation","target":"DALL-E 2","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-horse-rides-astronaut-c336ee-o1","statement_date":"2022-05-28","statement_date_precision":"day","excerpt":"DALL-E 2 is, I dare say, not as smart as it seems.","locator":"paragraph 18","document":{"url":"https://garymarcus.substack.com/p/horse-rides-astronaut","title":"Horse rides astronaut","publisher":"Marcus on AI","published_at":"2022-05-28T14:29:44Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:21Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"e4a8233fec988ccb75c5e8a45ac1ad325ab13313a43cd661957d6e57ebb49d48","receipts":1}],"first_statement_date":"2022-05-28","related":[],"current":{"assessment_id":"marcus-horse-rides-astronaut-c336ee-a1","payload":{"claim_id":"marcus-horse-rides-astronaut-c336ee","subject_id":"gary-marcus","normalized_proposition":"DALL-E 2 is not as smart as it appears.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"DALL-E 2 is, I dare say, not as smart as it seems.","locator":"paragraph 18","url":"https://garymarcus.substack.com/p/horse-rides-astronaut","date":"2022-05-28","document_title":"Horse rides astronaut","observed_at":"2026-10-29T00:05:21Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"cebee152ce57c2555ae1dac83518ea67cbbb148933f6ac355eab92ef10db3127","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:10:54Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-horse-rides-astronaut-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:10:54Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-horse-rides-astronaut-c336ee-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-d5bd09","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-horse-rides-astronaut-d5bd09-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-d5bd09#assessment-marcus-horse-rides-astronaut-d5bd09-a1"}],"claim":{"claim_id":"marcus-horse-rides-astronaut-d5bd09","subject_id":"gary-marcus","revision":1,"content_hash":"3b25bf2a7e9426cb99c3e2dfd2796f3667098a63df74417946b8cb96e1854364","normalized_proposition":"The machine learning community has recently made premature declarations and may not have learned the Clever Hans lesson about overclaiming.","kind":"observation","target":"the machine learning community","conditions":[],"original_horizon_text":null,"topic_tags":["other"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-horse-rides-astronaut-d5bd09-o1","statement_date":"2022-05-28","statement_date_precision":"day","excerpt":"Judging by all the premature declarations I have seen from the machine learning community of late, I am not sure AI has yet fully learned it.","locator":"paragraph 17","document":{"url":"https://garymarcus.substack.com/p/horse-rides-astronaut","title":"Horse rides astronaut","publisher":"Marcus on AI","published_at":"2022-05-28T14:29:44Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:21Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"e4a8233fec988ccb75c5e8a45ac1ad325ab13313a43cd661957d6e57ebb49d48","receipts":1}],"first_statement_date":"2022-05-28","related":[],"current":{"assessment_id":"marcus-horse-rides-astronaut-d5bd09-a1","payload":{"claim_id":"marcus-horse-rides-astronaut-d5bd09","subject_id":"gary-marcus","normalized_proposition":"The machine learning community has recently made premature declarations and may not have learned the Clever Hans lesson about overclaiming.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Judging by all the premature declarations I have seen from the machine learning community of late, I am not sure AI has yet fully learned it.","locator":"paragraph 17","url":"https://garymarcus.substack.com/p/horse-rides-astronaut","date":"2022-05-28","document_title":"Horse rides astronaut","observed_at":"2026-10-29T00:05:21Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"30a71a4b24c42d4b960ad1b8804f0e942e6f6aafe4c717169a76e229d92a2a2b","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T01:10:54Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-screen-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-horse-rides-astronaut-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T01:10:54Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-horse-rides-astronaut-d5bd09-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-dba2b0","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-horse-rides-astronaut-dba2b0-a3","status":"current","url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-dba2b0#assessment-marcus-horse-rides-astronaut-dba2b0-a3"}],"claim":{"claim_id":"marcus-horse-rides-astronaut-dba2b0","subject_id":"gary-marcus","revision":1,"content_hash":"80f75f7d0c82a0d114e008f057e86eda4f0a88ca876c1d786cc63784a121c463","normalized_proposition":"No one yet knows how to build a system that derives semantics of wholes from parts as a function of syntax, which is what is needed.","kind":"observation","target":"AI field","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-horse-rides-astronaut-dba2b0-o1","statement_date":"2022-05-28","statement_date_precision":"day","excerpt":"What one really needs, and no one yet knows how to build, is a system that can derive semantics of wholes from their parts as a function of their syntax.","locator":"paragraph 91","document":{"url":"https://garymarcus.substack.com/p/horse-rides-astronaut","title":"Horse rides astronaut","publisher":"Marcus on AI","published_at":"2022-05-28T14:29:44Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:21Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"e4a8233fec988ccb75c5e8a45ac1ad325ab13313a43cd661957d6e57ebb49d48","receipts":1}],"first_statement_date":"2022-05-28","related":[],"current":{"assessment_id":"marcus-horse-rides-astronaut-dba2b0-a3","payload":{"claim_id":"marcus-horse-rides-astronaut-dba2b0","subject_id":"gary-marcus","normalized_proposition":"No one yet knows how to build a system that derives semantics of wholes from parts as a function of syntax, which is what is needed.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"What one really needs, and no one yet knows how to build, is a system that can derive semantics of wholes from their parts as a function of their syntax.","locator":"paragraph 91","url":"https://garymarcus.substack.com/p/horse-rides-astronaut","date":"2022-05-28","document_title":"Horse rides astronaut","observed_at":"2026-10-29T00:05:21Z"},"criteria":{"from_speaker":["no one yet knows how to build","derive semantics of wholes from their parts as a function of their syntax","What one really needs"],"our_interpretation":["In context, the ordinary reading concerns general, reliable compositional language understanding of the kind needed by systems such as Imagen. It does not deny that restricted semantic parsers can compose meanings.","Relevant abilities include understanding unfamiliar combinations, distinguishing reversed participant roles, and preserving meaning across sensible paraphrases. These are contextual illustrations, not an added numerical benchmark.","The observation describes the field's state on May 28, 2022. I use that date as the reporting snapshot; earlier experiments provide background rather than independently establishing the entire field's state that day.","I assess the factual assertion about what researchers knew how to build. The phrase about what is needed supplies its practical context, rather than a separate claim that one particular architecture is necessary."],"support_condition":"A sufficiently comprehensive assessment of methods available by the post establishes that general, reliable composition of sentence meanings from parts and syntax remained unsolved. Failures in selected systems alone do not establish that nobody knew how to build such a system.","contradiction_condition":"A documented method available by May 28, 2022 demonstrates reliable, syntax-sensitive composition across varied ordinary language, including unfamiliar combinations and sensible paraphrases. Success confined to a restricted grammar or benchmark is insufficient by itself."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"October 2020 COGS experiments found weak and variable compositional generalization in the tested neural models [e1]. February 2022 research found that two compositional parsers maintained high accuracy across the tested COGS recursion depths [e2]. These results establish specific failures and successes, but neither a field-wide absence of reliable compositional understanding nor a method demonstrating that ability across varied ordinary language [e1, e2]. February 2020 research also identified distinct aspects of compositionality requiring separate tests [e3]. Together, these studies leave the field-wide claim about May 28, 2022 unresolved [e1, e2, e3].","limitations":["The studies do not comprehensively assess methods available on May 28, 2022 [e1, e2, e3].","The COGS results do not establish broad reliability across ordinary language, unfamiliar combinations, and sensible paraphrases [e1, e2]."],"evidence":[{"id":"e1","relation":"context","excerpt":"generalization accuracy","locator":"Section 1, Introduction, paragraph reporting COGS evaluation results","url":"https://arxiv.org/html/2010.05465v1","publisher":"arXiv; Najoung Kim and Tal Linzen","event_date":"2020-10","available_at":"2020-10-12","explanation":"The experiments document unstable compositional generalization in the tested architectures. They concern particular models and an earlier period, so they cannot establish Marcus's field-wide assertion."},{"id":"e2","relation":"context","excerpt":"high accuracy across all recursion depths.","locator":"Section 5, Depth generalization (recursion), discussion of Figure 4","url":"https://arxiv.org/html/2202.11937v1","publisher":"arXiv; Pia Weißenhorn, Yuekun Yao, Lucia Donatelli and Alexander Koller","event_date":"2022-02","available_at":"2022-02-24","explanation":"This demonstrates successful structural generalization across the tested COGS recursion depths. The paper also discusses broad-coverage parsing, making it material counterevidence to a blanket dismissal of compositional systems, but these evaluations do not establish general reliability across ordinary language."},{"id":"e3","relation":"context","excerpt":"different interpretations","locator":"Abstract, third sentence","url":"https://arxiv.org/abs/1908.08351v2","publisher":"arXiv; Dieuwke Hupkes, Verna Dankers, Mathijs Mul and Elia Bruni","event_date":"2020-02","available_at":"2020-02-23","explanation":"The research treats compositionality as requiring several distinct tests. It explains why a single successful or failed benchmark does not settle the broader contextual claim."}],"what_would_resolve":"A documented method available by May 28, 2022 demonstrating reliable, syntax-sensitive understanding across varied ordinary language, or a comprehensive historical assessment establishing that this capability remained unsolved then.","retrospective":true,"plain_reading":true},"payload_hash":"de85deb83c1a42f0c71f662ea1e28013b81d66bf8b329c53a9f20f85d26ba263","policy_version":"2026-10-28.7","recorded_at":"2026-10-31T20:01:09Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-dba2b0-1x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-horse-rides-astronaut-dba2b0-1x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-dba2b0-1x-final-anthropic-2"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e3","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-31T20:01:09Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-horse-rides-astronaut-dba2b0-assessment-published-a3","marcus-horse-rides-astronaut-dba2b0-held-a1"],"last_substantive_change_at":"2026-10-31T22:29:51Z"}},{"url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-f165b1","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-horse-rides-astronaut-f165b1-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-horse-rides-astronaut-f165b1#assessment-marcus-horse-rides-astronaut-f165b1-a2"}],"claim":{"claim_id":"marcus-horse-rides-astronaut-f165b1","subject_id":"gary-marcus","revision":2,"content_hash":"d6f4e432e46904bd160d632028428fd8b076dc63d3c6e48cafb1e3cb40022ce6","normalized_proposition":"Because the systems under discussion (Imagen, DALL-E 2 and similar architectures) represent phrases as strings and have no step to derive hierarchical syntactic structures, it is just wishful thinking to expect them to be fully sensitive to semantics as it relates to syntactic structure.","kind":"causal","target":"text-to-image systems","conditions":[{"text":"Imagen, DALL-E 2 and similar architectures under discussion","type":"scope","from_source":true},{"text":"phrases represented as strings, with no step to derive syntactic structures","type":"qualifier","from_source":true},{"text":"fully sensitive","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-horse-rides-astronaut-f165b1-o1","statement_date":"2022-05-28","statement_date_precision":"day","excerpt":"Furthermore, there is no step in the process to try to derive such structures; so it is just wishful thinking to expect them to be fully sensitive to semantics as it relates to syntactic structure.","locator":"paragraph 100","document":{"url":"https://garymarcus.substack.com/p/horse-rides-astronaut","title":"Horse rides astronaut","publisher":"Marcus on AI","published_at":"2022-05-28T14:29:44Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:21Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"e4a8233fec988ccb75c5e8a45ac1ad325ab13313a43cd661957d6e57ebb49d48","receipts":1}],"first_statement_date":"2022-05-28","related":[],"current":{"assessment_id":"marcus-horse-rides-astronaut-f165b1-a2","payload":{"claim_id":"marcus-horse-rides-astronaut-f165b1","subject_id":"gary-marcus","normalized_proposition":"Because the systems under discussion (Imagen, DALL-E 2 and similar architectures) represent phrases as strings and have no step to derive hierarchical syntactic structures, it is just wishful thinking to expect them to be fully sensitive to semantics as it relates to syntactic structure.","kind":"causal","conditions":["Imagen, DALL-E 2 and similar architectures under discussion","phrases represented as strings, with no step to derive syntactic structures","fully sensitive"],"original_horizon_text":null,"statement":{"excerpt":"Furthermore, there is no step in the process to try to derive such structures; so it is just wishful thinking to expect them to be fully sensitive to semantics as it relates to syntactic structure.","locator":"paragraph 100","url":"https://garymarcus.substack.com/p/horse-rides-astronaut","date":"2022-05-28","document_title":"Horse rides astronaut","observed_at":"2026-10-29T00:05:21Z"},"criteria":{"from_speaker":["no step in the process to try to derive such structures","so","fully sensitive to semantics","as it relates to syntactic structure"],"our_interpretation":["Read 'such structures' as explicit hierarchical syntactic representations, following the preceding reference to syntactic trees.","Assess the asserted causal connection between lacking an explicit structure-building step and lacking reliable syntax-dependent semantic behavior.","The named systems are the 2022 Imagen and DALL-E 2 architectures. Later or different models can inform the mechanism but cannot directly settle their performance.","'Fully sensitive' means systematic handling of meaning-changing syntactic distinctions, rather than occasional successful examples."],"support_condition":"Controlled experiments or a sufficiently specific theoretical result establish that the missing structure-building step explains the relevant semantic failures in these systems.","contradiction_condition":"Matched evidence demonstrates systematic syntax-dependent semantic reliability without such a step, or otherwise rules out its absence as the asserted cause."},"axis":"causal_assessment","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Imagen's May 2022 paper describes pretrained text encoders and diffusion conditioning (e1). Hewitt's April 4, 2019 demonstration shows recoverable syntactic relationships in BERT representations, but does not test Imagen or DALL-E 2 (e2). Neither item establishes or rules out the absence of explicit syntactic structure-building as the cause of semantic failures in the named image-generation systems (e1; e2). The causal claim remains unresolved on this evidence (e1; e2).","limitations":["The supplied architecture evidence concerns Imagen and does not independently document DALL-E 2's pipeline (e1).","Recoverable syntax in BERT does not establish reliable use of syntax during image generation (e2).","The packet contains no controlled intervention separating encoder, generation, training-data and sampling effects."],"evidence":[{"id":"e1","relation":"context","excerpt":"Imagen explores pretrained text encoders:","locator":"Section 2.1; conditioning architecture in sections 2 and 2.5","url":"https://arxiv.org/html/2205.11487v1","publisher":"Google Research researchers via arXiv","event_date":"2022-05","available_at":"2022-05-23","explanation":"The documented pipeline uses text embeddings to condition diffusion models and specifies no explicit tree-building stage. That supports the architectural premise under the explicit-parser reading, but does not establish its claimed causal consequence."},{"id":"e2","relation":"context","excerpt":"subjects and verbs close","locator":"A brief example on subject-verb number agreement","url":"https://www.cs.columbia.edu/~johnhew/structural-probe.html","publisher":"John Hewitt","event_date":"2019","available_at":"2019-04-04","explanation":"The author demonstrates recoverable syntactic relationships in learned BERT representations without an explicit parser in the underlying model. This cautions against inferring absence of syntactic information from architecture alone, but concerns a different system and does not prove full semantic reliability."}],"what_would_resolve":"Controlled experiments on the 2022 Imagen and DALL-E 2 systems, or applicable theory, establishing or ruling out the proposed cause. Experiments should manipulate syntactic structure-building while controlling other factors and systematically test matched prompts with changed roles or relations.","retrospective":true,"plain_reading":true},"payload_hash":"af2d691d5e5245b6107837e6391973a73508ed3244e22add8f9f6bde6e662485","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T22:02:33Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-2b3e87-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-horse-rides-astronaut-2b3e87-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-horse-rides-astronaut-f165b1-1x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"demonstration","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T22:02:33Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-horse-rides-astronaut-f165b1-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T22:52:30Z"}},{"url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-5ad6a3","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-noam-chomsky-and-gpt-3-5ad6a3-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-5ad6a3#assessment-marcus-noam-chomsky-and-gpt-3-5ad6a3-a1"}],"claim":{"claim_id":"marcus-noam-chomsky-and-gpt-3-5ad6a3","subject_id":"gary-marcus","revision":1,"content_hash":"dd8cc80f9af38bb4d36447c6c0b2a65ff000cb10a1218155fc49817ef78b92d1","normalized_proposition":"New AI systems with a firmer innate, Kantian basis might eventually tell us more about the human mind.","kind":"forecast","target":"nativist AI systems","conditions":[],"original_horizon_text":"eventually","topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-noam-chomsky-and-gpt-3-5ad6a3-o1","statement_date":"2022-05-21","statement_date_precision":"day","excerpt":"A new breed of AI systems, with a firmer innate basis, closer to the Kantian model, but out of step with current Zeitgeist, might eventually tell us more.","locator":"paragraph 26","document":{"url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","title":"Noam Chomsky and GPT-3","publisher":"Marcus on AI","published_at":"2022-05-21T17:45:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:23Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"07d5a897e1c3fa996767148162ac6fcb20d0ba1228f4392f98d810f139f1b769","receipts":1}],"first_statement_date":"2022-05-21","related":[],"current":{"assessment_id":"marcus-noam-chomsky-and-gpt-3-5ad6a3-a1","payload":{"claim_id":"marcus-noam-chomsky-and-gpt-3-5ad6a3","subject_id":"gary-marcus","normalized_proposition":"New AI systems with a firmer innate, Kantian basis might eventually tell us more about the human mind.","kind":"forecast","conditions":[],"original_horizon_text":"eventually","statement":{"excerpt":"A new breed of AI systems, with a firmer innate basis, closer to the Kantian model, but out of step with current Zeitgeist, might eventually tell us more.","locator":"paragraph 26","url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","date":"2022-05-21","document_title":"Noam Chomsky and GPT-3","observed_at":"2026-10-29T00:05:23Z"},"criteria":{"from_speaker":["eventually"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"231680ededea52772d34503f3cd4440c7ae819ac46a7e226cc8cd4c2b084753e","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-noam-chomsky-and-gpt-3-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-noam-chomsky-and-gpt-3-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-noam-chomsky-and-gpt-3-5ad6a3-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-5c52cd","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-noam-chomsky-and-gpt-3-5c52cd-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-5c52cd#assessment-marcus-noam-chomsky-and-gpt-3-5c52cd-a1"}],"claim":{"claim_id":"marcus-noam-chomsky-and-gpt-3-5c52cd","subject_id":"gary-marcus","revision":2,"content_hash":"dc8055bddb562c9ebc2ef75c90c77a60831c6b674ec430b86fca2e1e264a37a4","normalized_proposition":"Because large language models, unlike human minds, do not try to relate their language to what they know about the world, they just aren't telling us much about human minds.","kind":"causal","target":"large language models","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-noam-chomsky-and-gpt-3-5c52cd-o1","statement_date":"2022-05-21","statement_date_precision":"day","excerpt":"Human minds try to relate their language to what they know about the world; large language models don’t. Which means the latter just aren’t telling us much about the former.","locator":"paragraph 12","document":{"url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","title":"Noam Chomsky and GPT-3","publisher":"Marcus on AI","published_at":"2022-05-21T17:45:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:23Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"07d5a897e1c3fa996767148162ac6fcb20d0ba1228f4392f98d810f139f1b769","receipts":1}],"first_statement_date":"2022-05-21","related":[],"current":{"assessment_id":"marcus-noam-chomsky-and-gpt-3-5c52cd-a1","payload":{"claim_id":"marcus-noam-chomsky-and-gpt-3-5c52cd","subject_id":"gary-marcus","normalized_proposition":"Because large language models, unlike human minds, do not try to relate their language to what they know about the world, they just aren't telling us much about human minds.","kind":"causal","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Human minds try to relate their language to what they know about the world; large language models don’t. Which means the latter just aren’t telling us much about the former.","locator":"paragraph 12","url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","date":"2022-05-21","document_title":"Noam Chomsky and GPT-3","observed_at":"2026-10-29T00:05:23Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"b4c6e67710e1252551002e635f54da58f815c6d20d10614514329bd7ee457cdc","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:52:48Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-noam-chomsky-and-gpt-3-screen-anthropic-3"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-noam-chomsky-and-gpt-3-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:52:48Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-noam-chomsky-and-gpt-3-5c52cd-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-61b596","standing":"current","assessment_label":"Contradicted","versions":[{"assessment_id":"marcus-noam-chomsky-and-gpt-3-61b596-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-61b596#assessment-marcus-noam-chomsky-and-gpt-3-61b596-a1"}],"claim":{"claim_id":"marcus-noam-chomsky-and-gpt-3-61b596","subject_id":"gary-marcus","revision":2,"content_hash":"2c62e82ea82e0354d1456456eaa175ad25b25fcde312d1491e49dc7af6edd975","normalized_proposition":"Systems like GPT-3 do not gravitate towards human language, and they never develop a cohesive understanding of the world, no matter how much experience they have.","kind":"forecast","target":"GPT-3","conditions":[{"text":"systems like GPT-3","type":"scope","from_source":true},{"text":"no matter how much experience they have","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities","scaling"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-noam-chomsky-and-gpt-3-61b596-o1","statement_date":"2022-05-21","statement_date_precision":"day","excerpt":"They do not gravitate towards human language, and they never develop a cohesive undestanding of the world, no matter how much experience they have.","locator":"paragraph 23","document":{"url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","title":"Noam Chomsky and GPT-3","publisher":"Marcus on AI","published_at":"2022-05-21T17:45:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:23Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"07d5a897e1c3fa996767148162ac6fcb20d0ba1228f4392f98d810f139f1b769","receipts":1}],"first_statement_date":"2022-05-21","related":[],"current":{"assessment_id":"marcus-noam-chomsky-and-gpt-3-61b596-a1","payload":{"claim_id":"marcus-noam-chomsky-and-gpt-3-61b596","subject_id":"gary-marcus","normalized_proposition":"Systems like GPT-3 do not gravitate towards human language, and they never develop a cohesive understanding of the world, no matter how much experience they have.","kind":"forecast","conditions":["systems like GPT-3","no matter how much experience they have"],"original_horizon_text":null,"statement":{"excerpt":"They do not gravitate towards human language, and they never develop a cohesive undestanding of the world, no matter how much experience they have.","locator":"paragraph 23","url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","date":"2022-05-21","document_title":"Noam Chomsky and GPT-3","observed_at":"2026-10-29T00:05:23Z"},"criteria":{"from_speaker":["Systems like GPT-3","do not gravitate towards human language","never develop a cohesive undestanding of the world","no matter how much experience they have"],"our_interpretation":["I read this as two categorical claims about a model family, including a permanent limitation, without a deadline.","I interpret systems like GPT-3 to include GPT-style autoregressive transformer language models, including experimentally trained GPT-2 models. This does not restrict the claim to the released GPT-3 checkpoint.","The ordinary reading of gravitate towards human language is a learning preference for human-language patterns over appropriate artificial-language controls. It does not require an exclusively human learning capacity.","Cohesive understanding means reliable, generalizable tracking of objects, events and their spatial, temporal and causal relationships. Success in one restricted environment is insufficient.","A demonstrated counterexample to either categorical component contradicts the combined assertion. It does not establish that the other component is false.","No matter how much experience is a universal qualification, not a conditional antecedent requiring a particular training run."],"support_condition":"Evidence establishing both limitations throughout the stated model family and across additional training experience. Finite examples of failure can support present limitations but cannot establish never.","contradiction_condition":"Replicated controlled evaluations demonstrate a learning preference for human-language patterns in the stated model family, or demonstrate broad, consistent and generalizable world understanding without supplying a separate symbolic world model."},"axis":"forecast_outcome","outcome":"contradicted","horizon_state":"conclusive_event","evaluation_as_of":"2026-10-28","rationale":"The language-learning component has a controlled counterexample and independent corroboration. GPT-2 experiments show some preference for human-language patterns over altered alternatives. That contradicts the categorical absence of such a tendency under the stated reading of systems like GPT-3. The studies do not show that models learn exactly like humans. Hunter's criticism limits the stronger grammatical interpretation. The separate claim about never developing cohesive world understanding remains unsettled: finite failures cannot prove never, and restricted simulation successes do not establish broad understanding.","limitations":["The contradiction applies to the language-learning component of the combined assertion.","Including GPT-2 experiments within systems like GPT-3 is an explicit scope interpretation.","Preference for human-language patterns is weaker than a complete human-like language faculty.","Experimental dates are reported at year precision; availability dates identify the cited versions."],"evidence":[{"id":"e1","relation":"contradicts","excerpt":"struggles to learn","locator":"Abstract, core finding","url":"https://arxiv.org/abs/2401.06416","publisher":"arXiv","event_date":"2024","available_at":"2024-08-02","explanation":"Kallini and colleagues compared GPT-2 learning across English and systematically altered languages, including different training stages. The observed preference conflicts with the language-learning component under the stated family-level interpretation."},{"id":"e2","relation":"contradicts","excerpt":"human-like inductive biases","locator":"Abstract, final sentence","url":"https://arxiv.org/abs/2502.18795v2","publisher":"arXiv","event_date":"2025","available_at":"2025-05-31","explanation":"Yang and colleagues extended testing to 12 languages from four families and found partial discrimination between attested and impossible patterns. This independently corroborates a preference, while explicitly rejecting equivalence with human learning."},{"id":"e3","relation":"context","excerpt":"does not actually test","locator":"Discussion of the WordHop–NoHop comparison","url":"https://arxiv.org/html/2410.12271v1","publisher":"arXiv","event_date":"2024","available_at":"2024-10-16","explanation":"Hunter identifies a confound between counting rules and non-adjacent dependencies. This limits claims that the initial experiment demonstrates specifically human grammatical constraints, although that stronger conclusion is not required by the ordinary preference criterion."},{"id":"e4","relation":"context","excerpt":"incoherence creates fragility","locator":"Abstract, sentence on generalization failures","url":"https://arxiv.org/abs/2406.03689","publisher":"arXiv","event_date":"2024","available_at":"2024-11-10","explanation":"Vafa and colleagues find weaknesses in world-model coherence across game, logic and navigation settings. These findings support concern about tested systems but cannot establish failure at every future training scale."},{"id":"e5","relation":"context","excerpt":"implicit text-based world models.","locator":"Section 8, Conclusion","url":"https://arxiv.org/html/2512.18832v2","publisher":"arXiv","event_date":"2026","available_at":"2026-03-05","explanation":"Li and colleagues report coherent state tracking and useful simulation in five text environments after targeted training. The restricted environments and dependence on training coverage prevent treating this as decisive evidence of broad world understanding."}],"what_would_resolve":"Broad controlled evaluations of state tracking, causal reasoning and transfer across unfamiliar domains would address the remaining world-understanding component.","retrospective":true,"plain_reading":true},"payload_hash":"365fe04438ec3e9d356ef0854d5b342d6823d57d0bacd877ec6901cdc6f247a2","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T08:01:38Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-noam-chomsky-and-gpt-3-61b596-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-noam-chomsky-and-gpt-3-61b596-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-noam-chomsky-and-gpt-3-61b596-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e3","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e4","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e5","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T08:01:38Z","next_review_due_at":null,"horizon_now":"conclusive_event"},"event_ids":["marcus-noam-chomsky-and-gpt-3-61b596-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-960ed6","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-noam-chomsky-and-gpt-3-960ed6-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-960ed6#assessment-marcus-noam-chomsky-and-gpt-3-960ed6-a1"}],"claim":{"claim_id":"marcus-noam-chomsky-and-gpt-3-960ed6","subject_id":"gary-marcus","revision":2,"content_hash":"87e6b01a01fd61b140b9c6faeeba2f4dda8f721ce61c36cc510ce8ace03b9892","normalized_proposition":"Because large language models could glom onto any other data set (e.g. programming languages) just as well as human data, they tell us far too little about how humans have come to be what they are.","kind":"causal","target":"large language models","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-noam-chomsky-and-gpt-3-960ed6-o1","statement_date":"2022-05-21","statement_date_precision":"day","excerpt":"Large language models can, parasitically, glom onto human data, but they could glom onto any other data set just as well (e.g. programming languages); as such, they tell us far too little about how humans have come to be what they are.","locator":"paragraph 42","document":{"url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","title":"Noam Chomsky and GPT-3","publisher":"Marcus on AI","published_at":"2022-05-21T17:45:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:23Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"07d5a897e1c3fa996767148162ac6fcb20d0ba1228f4392f98d810f139f1b769","receipts":1}],"first_statement_date":"2022-05-21","related":[],"current":{"assessment_id":"marcus-noam-chomsky-and-gpt-3-960ed6-a1","payload":{"claim_id":"marcus-noam-chomsky-and-gpt-3-960ed6","subject_id":"gary-marcus","normalized_proposition":"Because large language models could glom onto any other data set (e.g. programming languages) just as well as human data, they tell us far too little about how humans have come to be what they are.","kind":"causal","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Large language models can, parasitically, glom onto human data, but they could glom onto any other data set just as well (e.g. programming languages); as such, they tell us far too little about how humans have come to be what they are.","locator":"paragraph 42","url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","date":"2022-05-21","document_title":"Noam Chomsky and GPT-3","observed_at":"2026-10-29T00:05:23Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"2ad5af021aeeb8e1eb42e3bb83d0fcaf8219d9fd631f9396bce2f0ce350567c2","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:52:48Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-noam-chomsky-and-gpt-3-screen-anthropic-3"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-noam-chomsky-and-gpt-3-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:52:48Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-noam-chomsky-and-gpt-3-960ed6-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-c476d0","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-noam-chomsky-and-gpt-3-c476d0-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-c476d0#assessment-marcus-noam-chomsky-and-gpt-3-c476d0-a1"}],"claim":{"claim_id":"marcus-noam-chomsky-and-gpt-3-c476d0","subject_id":"gary-marcus","revision":2,"content_hash":"61183b508574b75b13ea285a234c790f3c94fd6b95c5d32966e0ec948cf9a974","normalized_proposition":"Because systems like GPT-3 don't tell us why human languages have the special character that they do, they have little explanatory value.","kind":"causal","target":"GPT-3","conditions":[{"text":"systems like GPT-3","type":"scope","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-noam-chomsky-and-gpt-3-c476d0-o1","statement_date":"2022-05-21","statement_date_precision":"day","excerpt":"Systems like GPT-3 don’t tell us why human languages have the special character that they do. As such, there is little explanatory value.","locator":"paragraph 15","document":{"url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","title":"Noam Chomsky and GPT-3","publisher":"Marcus on AI","published_at":"2022-05-21T17:45:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:23Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"07d5a897e1c3fa996767148162ac6fcb20d0ba1228f4392f98d810f139f1b769","receipts":1}],"first_statement_date":"2022-05-21","related":[],"current":{"assessment_id":"marcus-noam-chomsky-and-gpt-3-c476d0-a1","payload":{"claim_id":"marcus-noam-chomsky-and-gpt-3-c476d0","subject_id":"gary-marcus","normalized_proposition":"Because systems like GPT-3 don't tell us why human languages have the special character that they do, they have little explanatory value.","kind":"causal","conditions":["systems like GPT-3"],"original_horizon_text":null,"statement":{"excerpt":"Systems like GPT-3 don’t tell us why human languages have the special character that they do. As such, there is little explanatory value.","locator":"paragraph 15","url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","date":"2022-05-21","document_title":"Noam Chomsky and GPT-3","observed_at":"2026-10-29T00:05:23Z"},"criteria":{"from_speaker":["systems like GPT-3"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"c343c5dbcdd80b359ffa2adba74783defaedc946bc5a0a092595649f10ecef18","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:52:48Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-noam-chomsky-and-gpt-3-screen-anthropic-3"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-noam-chomsky-and-gpt-3-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:52:48Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-noam-chomsky-and-gpt-3-c476d0-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-c49391","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-noam-chomsky-and-gpt-3-c49391-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-c49391#assessment-marcus-noam-chomsky-and-gpt-3-c49391-a1"}],"claim":{"claim_id":"marcus-noam-chomsky-and-gpt-3-c49391","subject_id":"gary-marcus","revision":1,"content_hash":"0ad4fb8b12a36125fdc9578b19b042852e13c615c618fa3d16a91c6f214531d2","normalized_proposition":"For AI to understand language, it should process language through built-in commonsense understanding instead of starting as a blank slate and learning statistics.","kind":"normative","target":"AI systems","conditions":[{"text":"if AI's are to understand language","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-noam-chomsky-and-gpt-3-c49391-o1","statement_date":"2022-05-21","statement_date_precision":"day","excerpt":"Humans know these things; if AI’s are to understand language, they should refract language through this sort basic understanding of common sense, rather than needlessly starting as blank slates and acquiring statistics without concepts.","locator":"paragraph 25","document":{"url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","title":"Noam Chomsky and GPT-3","publisher":"Marcus on AI","published_at":"2022-05-21T17:45:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:23Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"07d5a897e1c3fa996767148162ac6fcb20d0ba1228f4392f98d810f139f1b769","receipts":1}],"first_statement_date":"2022-05-21","related":[],"current":{"assessment_id":"marcus-noam-chomsky-and-gpt-3-c49391-a1","payload":{"claim_id":"marcus-noam-chomsky-and-gpt-3-c49391","subject_id":"gary-marcus","normalized_proposition":"For AI to understand language, it should process language through built-in commonsense understanding instead of starting as a blank slate and learning statistics.","kind":"normative","conditions":["if AI's are to understand language"],"original_horizon_text":null,"statement":{"excerpt":"Humans know these things; if AI’s are to understand language, they should refract language through this sort basic understanding of common sense, rather than needlessly starting as blank slates and acquiring statistics without concepts.","locator":"paragraph 25","url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","date":"2022-05-21","document_title":"Noam Chomsky and GPT-3","observed_at":"2026-10-29T00:05:23Z"},"criteria":{"from_speaker":["if AI's are to understand language"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"1b9c1b324bf2ee9ee53454040cc49733a43822d9266d9de02d23116d69219728","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-noam-chomsky-and-gpt-3-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-noam-chomsky-and-gpt-3-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-noam-chomsky-and-gpt-3-c49391-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-d8d192","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-noam-chomsky-and-gpt-3-d8d192-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-d8d192#assessment-marcus-noam-chomsky-and-gpt-3-d8d192-a1"}],"claim":{"claim_id":"marcus-noam-chomsky-and-gpt-3-d8d192","subject_id":"gary-marcus","revision":1,"content_hash":"43c5ebfa6ccdb545d5bf3a45defb6dc15e05251d7639900cd7c5e1e30d6cc37b","normalized_proposition":"LLMs' failures argue strongly for exploring Kantian AI systems that start with built-in representations of space, time, causality and some universal grammar.","kind":"normative","target":"AI research","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-noam-chomsky-and-gpt-3-d8d192-o1","statement_date":"2022-05-21","statement_date_precision":"day","excerpt":"They are an embarrassment to Locke’s conjecture, and a strong argument for exploring their very antithesis, which would be Kantian systems that start with representations of space and time and causality, and at least some kernels of universal grammar.","locator":"paragraph 23","document":{"url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","title":"Noam Chomsky and GPT-3","publisher":"Marcus on AI","published_at":"2022-05-21T17:45:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:23Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"07d5a897e1c3fa996767148162ac6fcb20d0ba1228f4392f98d810f139f1b769","receipts":1}],"first_statement_date":"2022-05-21","related":[],"current":{"assessment_id":"marcus-noam-chomsky-and-gpt-3-d8d192-a1","payload":{"claim_id":"marcus-noam-chomsky-and-gpt-3-d8d192","subject_id":"gary-marcus","normalized_proposition":"LLMs' failures argue strongly for exploring Kantian AI systems that start with built-in representations of space, time, causality and some universal grammar.","kind":"normative","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"They are an embarrassment to Locke’s conjecture, and a strong argument for exploring their very antithesis, which would be Kantian systems that start with representations of space and time and causality, and at least some kernels of universal grammar.","locator":"paragraph 23","url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","date":"2022-05-21","document_title":"Noam Chomsky and GPT-3","observed_at":"2026-10-29T00:05:23Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"d2d97e20f21ab48907d94bbae1a1ad953625e7fbe209bbe10c1ac8566f94671f","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-noam-chomsky-and-gpt-3-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-noam-chomsky-and-gpt-3-screen-openai-1"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-noam-chomsky-and-gpt-3-d8d192-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-f648bc","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-noam-chomsky-and-gpt-3-f648bc-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-noam-chomsky-and-gpt-3-f648bc#assessment-marcus-noam-chomsky-and-gpt-3-f648bc-a1"}],"claim":{"claim_id":"marcus-noam-chomsky-and-gpt-3-f648bc","subject_id":"gary-marcus","revision":1,"content_hash":"ee5e9456c1bae3baea7524b27fadd8094ae0eb790781414d8348c839ad8ed9a6","normalized_proposition":"GPT-3 cannot avoid producing misinformation because it models how words relate to each other, not how language relates to the world.","kind":"causal","target":"GPT-3","conditions":[],"original_horizon_text":null,"topic_tags":["reliability","safety"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-noam-chomsky-and-gpt-3-f648bc-o1","statement_date":"2022-05-21","statement_date_precision":"day","excerpt":"There is no intention to create misinformation—but also no capacity to avoid it, because fundamentally, GPT-3 is a model of how words relate to one another, not a model of how language might relate to the perceived world.","locator":"paragraph 12","document":{"url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","title":"Noam Chomsky and GPT-3","publisher":"Marcus on AI","published_at":"2022-05-21T17:45:07Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:23Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"07d5a897e1c3fa996767148162ac6fcb20d0ba1228f4392f98d810f139f1b769","receipts":1}],"first_statement_date":"2022-05-21","related":[],"current":{"assessment_id":"marcus-noam-chomsky-and-gpt-3-f648bc-a1","payload":{"claim_id":"marcus-noam-chomsky-and-gpt-3-f648bc","subject_id":"gary-marcus","normalized_proposition":"GPT-3 cannot avoid producing misinformation because it models how words relate to each other, not how language relates to the world.","kind":"causal","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"There is no intention to create misinformation—but also no capacity to avoid it, because fundamentally, GPT-3 is a model of how words relate to one another, not a model of how language might relate to the perceived world.","locator":"paragraph 12","url":"https://garymarcus.substack.com/p/noam-chomsky-and-gpt-3","date":"2022-05-21","document_title":"Noam Chomsky and GPT-3","observed_at":"2026-10-29T00:05:23Z"},"criteria":{"from_speaker":["no intention to create misinformation","but also no capacity to avoid it","because fundamentally, GPT-3 is a model of how words relate to one another","not a model of how language might relate to the perceived world"],"our_interpretation":["Two parts. (a) GPT-3 had no reliable built-in way to avoid producing false content. (b) The cause is that it is trained only on relations among words, without grounding in the perceived world.","Read fairly, 'no capacity to avoid it' means it cannot reliably avoid misinformation, not that it never says anything true."],"support_condition":"Controlled evidence that GPT-3-era models' misinformation stems from text-only training, and that grounding in the world is what would be needed to fix it.","contradiction_condition":"Evidence that GPT-3-era models without world grounding could reliably avoid misinformation, or that the errors had a different main cause."},"axis":"causal_assessment","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The claim has an empirical part and a causal part. The evidence fits the empirical part read fairly. GPT-3 and InstructGPT kept making up information, at 41% and 21% on closed-domain tasks (e2). But fine-tuning without any grounding in the world halved that rate, which shows some capacity to reduce misinformation. TruthfulQA's authors tie false answers to imitating training text (e1), which partly fits Marcus's story. No cited study isolates the cause he names, being a model of word relations rather than of the world, from other factors. The causal part therefore stays open on the available evidence.","limitations":["The evidence is about error rates, not about mechanism.","Settling this would need a controlled grounded-versus-ungrounded comparison, which no source here provides."],"evidence":[{"id":"e1","relation":"context","excerpt":"However, this result is expected if false answers are learned from the training distribution.","locator":"Abstract","url":"https://arxiv.org/abs/2109.07958","publisher":"ACL 2022 (Lin, Hilton, Evans), via arXiv","event_date":"2021-09","available_at":"2021-09-08","explanation":"Links falsehoods to imitating training text, which partly fits Marcus's word-statistics story. It does not test whether grounding in the world is the missing factor."},{"id":"e2","relation":"context","excerpt":"InstructGPT models make up information not present in the input about half as often as GPT-3 (a\n21% vs. 41% hallucination rate, respectively).","locator":"Section 1 Introduction, summary of findings","url":"https://arxiv.org/pdf/2203.02155","publisher":"arXiv (OpenAI authors)","event_date":"2022-03","available_at":"2022-03-04","explanation":"Human-feedback fine-tuning, with no grounding in the world, halved fabrication but left it at 21%. This fits 'cannot reliably avoid' and also shows some capacity to reduce errors, so the causal story stays unsettled."}],"what_would_resolve":"Controlled studies comparing text-only and world-grounded models of similar scale on misinformation rates, or mechanistic work showing why GPT-3-era models fabricate.","retrospective":true,"plain_reading":true},"payload_hash":"0806e439dcf4d61d7983acdaccc30f4b9a89a7470ec219625c4c92fd69166df7","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T08:01:38Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-noam-chomsky-and-gpt-3-61b596-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-noam-chomsky-and-gpt-3-61b596-4x-review-openai-1"}],"final_text":{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-noam-chomsky-and-gpt-3-61b596-4x-final-openai-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T08:01:38Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-noam-chomsky-and-gpt-3-f648bc-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-0f409d","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-0f409d-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-0f409d#assessment-marcus-the-new-science-of-alt-intelligence-0f409d-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-0f409d","subject_id":"gary-marcus","revision":2,"content_hash":"33b1a4f8dfcaa49da115a91f92e45179183b698c2d34448f2fe50ea8af800eb8","normalized_proposition":"Virtually none of the critical AI problems that cognitive scientists such as Steve Pinker, Judea Pearl, Jerry Fodor and Marcus have been pointing to for decades is actually solved yet.","kind":"observation","target":"AI","conditions":[{"text":"virtually none","type":"scope","from_source":true},{"text":"yet","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-0f409d-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"Virtually none of the critical problems for AI that cognitive scientists such as Steve Pinker, Judea Pearl, the late Jerry Fodor, and myself have been pointing to for decades is actually solved yet.","locator":"paragraph 57","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-0f409d-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-0f409d","subject_id":"gary-marcus","normalized_proposition":"Virtually none of the critical AI problems that cognitive scientists such as Steve Pinker, Judea Pearl, Jerry Fodor and Marcus have been pointing to for decades is actually solved yet.","kind":"observation","conditions":["virtually none","yet"],"original_horizon_text":null,"statement":{"excerpt":"Virtually none of the critical problems for AI that cognitive scientists such as Steve Pinker, Judea Pearl, the late Jerry Fodor, and myself have been pointing to for decades is actually solved yet.","locator":"paragraph 57","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":["Virtually none","critical problems for AI","Steve Pinker, Judea Pearl, the late Jerry Fodor, and myself","pointing to for decades","actually solved yet"],"our_interpretation":["The immediate context identifies general text comprehension, modeling speaker intent, general reasoning and compositionality as examples of the problems.","Solved means reliable competence across the relevant domain, rather than success on selected demonstrations or a narrow benchmark.","Those examples guide the reading but cannot silently replace the larger, unspecified collection attributed to the named researchers.","The target is May 2022 up to publication, the latest partial calendar month for this ongoing state. Older research is context."],"support_condition":"A defensible inventory of the referenced problems and contemporary assessments establish that nearly all remained unsolved under their original scope.","contradiction_condition":"Contemporary evidence establishes reliable solutions to a substantial portion of that inventory."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The research supports caution about equating benchmark performance with solved general capabilities, while also documenting progress. It does not establish which problems belong in Marcus's collection or how many were solved. Restricting the claim to the next sentence's examples would make it easier to assess but would narrow his actual wording.","limitations":["A complete inventory cannot be inferred from the researchers' names alone.","Neither isolated successes nor isolated failures settle the proportion claimed."],"evidence":[{"id":"e1","relation":"context","excerpt":"casts doubts","locator":"Section 1, Introduction, second paragraph","url":"https://www.cambridge.org/core/journals/natural-language-engineering/article/survey-of-methods-for-revealing-and-overcoming-weaknesses-of-datadriven-natural-language-understanding/BC8EBADC7D8E5CFC67FC1FE5958E13CC","publisher":"Cambridge University Press","event_date":"2022-04","available_at":"2022-04-22","explanation":"The survey explains why strong benchmark results need not establish robust language understanding. It supports caution about declaring problems solved, but does not count the problems Marcus references."},{"id":"e2","relation":"context","excerpt":"promise","locator":"Section 8, Conclusions","url":"https://arxiv.org/pdf/2205.06175v1","publisher":"DeepMind / arXiv","event_date":"2022-05","available_at":"2022-05-12","explanation":"The authors report progress toward learning beyond trained tasks. This qualifies an overly sweeping reading of Marcus, but promising transfer is not a demonstrated solution to the broader problems."}],"what_would_resolve":"An inventory grounded in the named researchers' original problem statements, followed by a problem-by-problem assessment of systems existing by May 14, 2022.","retrospective":true,"plain_reading":true},"payload_hash":"93b2c664056c0998bb4f36c3beb3c36e6ba2fa7c4a524b5714dc96f124b46426","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T21:59:19Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-0f409d-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-0f409d-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-0f409d-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T21:59:19Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-0f409d-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T22:52:30Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-11b850","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-11b850-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-11b850#assessment-marcus-the-new-science-of-alt-intelligence-11b850-a2"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-11b850","subject_id":"gary-marcus","revision":2,"content_hash":"0a921f013d37c361ccddd49271142df8cdfe121826a4edd09ca397ce4c6c9e6f","normalized_proposition":"If we are to build AGI, we will need to learn something from humans: how they reason and understand the physical world, and how they represent and acquire language and complex concepts.","kind":"conditional_forecast","target":"AGI","conditions":[{"text":"if we are to build AGI","type":"antecedent","from_source":true},{"text":"learn something from humans","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["agi-timelines"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-11b850-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"The bottom line is this, something that AI once cherished but has now forgotten: If we are to build AGI, we are going to need to learn something from humans, how they reason and understand the physical world, and how they represent and acquire language and complex concepts.","locator":"paragraph 72","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-11b850-a2","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-11b850","subject_id":"gary-marcus","normalized_proposition":"If we are to build AGI, we will need to learn something from humans: how they reason and understand the physical world, and how they represent and acquire language and complex concepts.","kind":"conditional_forecast","conditions":["if we are to build AGI","learn something from humans"],"original_horizon_text":null,"statement":{"excerpt":"The bottom line is this, something that AI once cherished but has now forgotten: If we are to build AGI, we are going to need to learn something from humans, how they reason and understand the physical world, and how they represent and acquire language and complex concepts.","locator":"paragraph 72","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":["If we are to build AGI","we are going to need to learn something from humans","how they reason and understand the physical world","how they represent and acquire language and complex concepts"],"our_interpretation":["Read this as a necessary-condition prediction about building AGI, rather than merely advice.","The antecedent is an established AGI achievement. Learning from humans is the proposed necessary requirement, not an additional antecedent that must be assumed.","Use broadly human-level, general cognitive competence as the ordinary meaning of AGI; a company declaration or one benchmark score alone does not establish it.","Interpret learning from humans as substantive design insights from human cognition. Exposure to human-written training data alone does not establish that requirement.","A successful system using cognitive insights would show compatibility with the claim, but would not by itself prove those insights were necessary."],"support_condition":"An independently established AGI achievement is accompanied by credible evidence that substantive insights from human reasoning, world understanding, language, or concept acquisition were necessary to its development.","contradiction_condition":"An independently established AGI system is developed without those substantive cognitive insights, with sufficiently complete development records to establish the counterexample."},"axis":"forecast_outcome","outcome":"unresolved","horizon_state":"open_horizon","evaluation_as_of":"2026-10-28","rationale":"ARC Prize's assessment, available September 3, 2026, describes Astra as progress toward generalization but explicitly declines to call it AGI [e7]. The AGI Society's September 14, 2026 update says there is no generally accepted empirical test for AGI and identifies gaps in the capability evidence [e8]. These assessments establish neither a qualifying AGI achievement nor the necessity of human cognitive insights, leaving the claim unresolved [e7, e8].","limitations":["These assessments do not establish that AGI has not been achieved elsewhere [e7, e8].","The evidence provides no qualifying system with an auditable development history sufficient to test the asserted necessity [e7, e8]."],"evidence":[{"id":"e7","relation":"context","excerpt":"not claiming","locator":"Section ARC-AGI Series","url":"https://arcprize.org/blog/astra","publisher":"ARC Prize","event_date":"2026-09","available_at":"2026-09-03","explanation":"The benchmark evaluator reports substantial progress but explicitly declines to treat it as an AGI achievement. This leaves the antecedent unestablished; it does not prove that AGI has not been built elsewhere."},{"id":"e8","relation":"context","excerpt":"There is no generally accepted empirical test for AGI.","locator":"Section Proposed tests of general intelligence","url":"https://agi-society.org/publications/evaluating-agi-claims","publisher":"AGI Society","event_date":"2026-09","available_at":"2026-09-14","explanation":"This assessment considers public declarations that AGI has arrived but concludes that the published evidence does not establish the achievement. September 14 is the disclosed update date of the version read; the assessment does not establish Marcus's proposed necessary development method."}],"what_would_resolve":"An independently established AGI achievement with auditable development records. A system developed without substantive insights from human cognition would contradict the claim. Support would require strong evidence that those insights were necessary, beyond showing that they were used.","retrospective":true,"plain_reading":true},"payload_hash":"b4bb480654e91dce796562a345c531175e82469a4f3416b91f8ce6887d754ab6","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:42:08Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-11b850-2x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-11b850-2x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-11b850-2x-final-anthropic-2"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e7","source_type":"benchmark","audited":null,"derived":null,"metric":null},{"id":"e8","source_type":"secondary_summary","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:42:08Z","next_review_due_at":null,"horizon_now":"open_horizon"},"event_ids":["marcus-the-new-science-of-alt-intelligence-11b850-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-174e93","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-174e93-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-174e93#assessment-marcus-the-new-science-of-alt-intelligence-174e93-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-174e93","subject_id":"gary-marcus","revision":1,"content_hash":"4f4445964373ece4edcd98724ce93f273b7cebababaaba07dbf440f4cec53113","normalized_proposition":"AI is still at an early stage of development.","kind":"observation","target":"AI field","conditions":[],"original_horizon_text":null,"topic_tags":["agi-timelines"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-174e93-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"We are in early days in AI.","locator":"paragraph 57","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-174e93-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-174e93","subject_id":"gary-marcus","normalized_proposition":"AI is still at an early stage of development.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"We are in early days in AI.","locator":"paragraph 57","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"24a5e02c159b07b841c3f5ed52722306f4a69889d3bc4ef559bab9e583fe30d2","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-174e93-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-1ae1e2","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-1ae1e2-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-1ae1e2#assessment-marcus-the-new-science-of-alt-intelligence-1ae1e2-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-1ae1e2","subject_id":"gary-marcus","revision":2,"content_hash":"2a6a9bc7f8c0688a12618297db9e510947d3334d631467b598652335045ade4a","normalized_proposition":"Anyone who is candid will recognise that these kinds of errors by current AI systems reveal that something is, for now, deeply amiss.","kind":"observation","target":"current AI systems","conditions":[{"text":"for now","type":"qualifier","from_source":true},{"text":"anyone who is candid","type":"scope","from_source":true}],"original_horizon_text":null,"topic_tags":["reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-1ae1e2-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"But anyone who is candid will recognize that these kinds of errors reveal that something is, for now, deeply amiss.","locator":"paragraph 34","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-1ae1e2-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-1ae1e2","subject_id":"gary-marcus","normalized_proposition":"Anyone who is candid will recognise that these kinds of errors by current AI systems reveal that something is, for now, deeply amiss.","kind":"observation","conditions":["for now","anyone who is candid"],"original_horizon_text":null,"statement":{"excerpt":"But anyone who is candid will recognize that these kinds of errors reveal that something is, for now, deeply amiss.","locator":"paragraph 34","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":["for now","anyone who is candid"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this too vague to test: informed readers would test it in different ways and reach different answers.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading"]},"payload_hash":"210c4e46cb24283972851d2dd940ea48ef6e665e771d4e57518121d43f395e78","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T03:08:15Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-screen-anthropic-3"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-screen-openai-3"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T03:08:15Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-1ae1e2-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-1cffad","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-1cffad-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-1cffad#assessment-marcus-the-new-science-of-alt-intelligence-1cffad-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-1cffad","subject_id":"gary-marcus","revision":1,"content_hash":"a9cd46a760a67bf811719ca0b73d717d95cf6bd6898ec802db5ef358582799dc","normalized_proposition":"The AI field should pursue multiple research directions rather than dismissing undeveloped ideas prematurely.","kind":"normative","target":"AI field","conditions":[],"original_horizon_text":null,"topic_tags":["other"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-1cffad-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"Let us all encourage a field that is open-minded enough to work in multiple directions, without prematurely dismissing ideas that happen to be not yet fully developed.","locator":"paragraph 68","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-1cffad-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-1cffad","subject_id":"gary-marcus","normalized_proposition":"The AI field should pursue multiple research directions rather than dismissing undeveloped ideas prematurely.","kind":"normative","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Let us all encourage a field that is open-minded enough to work in multiple directions, without prematurely dismissing ideas that happen to be not yet fully developed.","locator":"paragraph 68","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"276fc3861db494854edead76950255ac0e8aeb4b3a654d2d62297ed1e5dbd484","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-1cffad-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-2b01c1","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-2b01c1-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-2b01c1#assessment-marcus-the-new-science-of-alt-intelligence-2b01c1-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-2b01c1","subject_id":"gary-marcus","revision":1,"content_hash":"05d90b035989c190070f445517028c6938ee8ccf459c0f5f57a64bb95c12561a","normalized_proposition":"The best path to AGI may not be through data-driven Alt Intelligence approaches.","kind":"forecast","target":"Alt Intelligence / scaling","conditions":[],"original_horizon_text":null,"topic_tags":["agi-timelines","scaling"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-2b01c1-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"It may just be that the best path to artificial (general) intelligence isn’t through Alt Intelligence, after all.","locator":"paragraph 68","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-2b01c1-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-2b01c1","subject_id":"gary-marcus","normalized_proposition":"The best path to AGI may not be through data-driven Alt Intelligence approaches.","kind":"forecast","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"It may just be that the best path to artificial (general) intelligence isn’t through Alt Intelligence, after all.","locator":"paragraph 68","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"bd85d32899637cbc350ed090a48c7e0430195da265a45bc58235912b7b7b583d","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-2b01c1-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-405ffd","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-405ffd-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-405ffd#assessment-marcus-the-new-science-of-alt-intelligence-405ffd-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-405ffd","subject_id":"gary-marcus","revision":2,"content_hash":"e6e59160ab138065a30a95134acc5d65dfe51db2183c739127cfbecaf1ad071d","normalized_proposition":"A large fraction of current AI researchers, or at least those currently in power, do not, so far as Marcus can tell, care about AI taking inspiration from natural intelligence.","kind":"observation","target":"current AI researchers","conditions":[{"text":"a large fraction of current AI researchers, or at least those currently in power","type":"scope","from_source":true},{"text":"so far as I can tell","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["other"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-405ffd-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"A large fraction of current AI researchers, or at least those currently in power, don’t (so far as I can tell) give a damn about any of this.","locator":"paragraph 2","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-405ffd-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-405ffd","subject_id":"gary-marcus","normalized_proposition":"A large fraction of current AI researchers, or at least those currently in power, do not, so far as Marcus can tell, care about AI taking inspiration from natural intelligence.","kind":"observation","conditions":["a large fraction of current AI researchers, or at least those currently in power","so far as I can tell"],"original_horizon_text":null,"statement":{"excerpt":"A large fraction of current AI researchers, or at least those currently in power, don’t (so far as I can tell) give a damn about any of this.","locator":"paragraph 2","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":["a large fraction of current AI researchers, or at least those currently in power","so far as I can tell"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found that evidence could not settle it: one found it too vague to test: informed readers would test it in different ways and reach different answers; the other found it about something no public evidence could show, such as a private intention or a counterfactual.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["no_plain_reading","no_public_evidence"]},"payload_hash":"7d69840d595d1ee9a121322ec0184fbff0a946eed2a422b3d973379d12b41911","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T03:08:15Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-screen-anthropic-3"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-screen-openai-3"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T03:08:15Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-405ffd-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-4a9f0c","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-4a9f0c-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-4a9f0c#assessment-marcus-the-new-science-of-alt-intelligence-4a9f0c-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-4a9f0c","subject_id":"gary-marcus","revision":1,"content_hash":"324921b7f57a24d23a49c87a58523634b005de74bc3fda058d26e96178d4c189","normalized_proposition":"AI research should draw clues from human and animal cognition without slavishly replicating human intelligence.","kind":"normative","target":"AI research","conditions":[],"original_horizon_text":null,"topic_tags":["other"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-4a9f0c-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"AI should certainly not be a slavish replica of human intelligence (which after all is flawed in its own ways, saddled with lousy memory and cognitive bias). But it should look to human (and animal cognition) for clues.","locator":"paragraph 71","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-4a9f0c-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-4a9f0c","subject_id":"gary-marcus","normalized_proposition":"AI research should draw clues from human and animal cognition without slavishly replicating human intelligence.","kind":"normative","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"AI should certainly not be a slavish replica of human intelligence (which after all is flawed in its own ways, saddled with lousy memory and cognitive bias). But it should look to human (and animal cognition) for clues.","locator":"paragraph 71","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"155fd845c4fad12d45d651afa5dbc87fb71ac1f614516f45fa081872d9b7c3a3","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-4a9f0c-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-75a6f1","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-75a6f1-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-75a6f1#assessment-marcus-the-new-science-of-alt-intelligence-75a6f1-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-75a6f1","subject_id":"gary-marcus","revision":2,"content_hash":"9fd9d1df58faaf722c1ede770b01ce71ed72fe306848a1f4829d64045e2da5a0","normalized_proposition":"Systems like DALL-E 2, GPT-3, Flamingo and Gato appear mind-blowing, but nobody who has looked at them carefully would confuse them for human intelligence.","kind":"observation","target":"DALL-E 2, GPT-3, Flamingo, Gato","conditions":[{"text":"who has looked at them carefully","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["capabilities"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-75a6f1-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"Systems like DALL-E 2, GPT-3, Flamingo, and Gato appear to be mind-blowing, but nobody who has looked at them carefully would confuse them for human intelligence.","locator":"paragraph 17","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-75a6f1-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-75a6f1","subject_id":"gary-marcus","normalized_proposition":"Systems like DALL-E 2, GPT-3, Flamingo and Gato appear mind-blowing, but nobody who has looked at them carefully would confuse them for human intelligence.","kind":"observation","conditions":["who has looked at them carefully"],"original_horizon_text":null,"statement":{"excerpt":"Systems like DALL-E 2, GPT-3, Flamingo, and Gato appear to be mind-blowing, but nobody who has looked at them carefully would confuse them for human intelligence.","locator":"paragraph 17","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":["DALL-E 2, GPT-3, Flamingo, and Gato","nobody who has looked at them carefully","would confuse them for human intelligence"],"our_interpretation":["The ordinary contextual reading concerns broad, flexible human intelligence, rather than human-level performance on a particular task.","The statement also asserts agreement among careful observers; demonstrations of errors do not independently establish that agreement.","I use April 1 through May 14, 2022, as the latest public evaluation cycle surrounding the systems discussed. This is an ongoing observation, not a claim about subsequent versions.","Mind-blowing is an expression of admiration and is not separately scored."],"support_condition":"Careful contemporaneous evaluations covering all four systems establish a clear distinction from broad human intelligence and substantiate the asserted agreement among informed observers.","contradiction_condition":"A documented careful contemporary evaluator regards a named system as exhibiting broadly equivalent human intelligence, or a suitable evaluation establishes such equivalence."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The records document important limitations, and a contemporary GPT-3 analysis also credits meaningful semantic abilities. Neither finding settles Marcus's broader assertion about human intelligence or what every careful observer would conclude. Scoring it as supported would require replacing the asserted agreement with the narrower claim that the systems made revealing errors.","limitations":["The literal and rhetorical readings of nobody require different evidence.","The quoted GPT-3 evidence comes from the article's readable abstract.","Task-specific failures are not a complete measure of human intelligence."],"evidence":[{"id":"e1","relation":"context","excerpt":"examples of such behaviours","locator":"Section 6.1, discussion of language-model limitations","url":"https://arxiv.org/html/2204.14198v1","publisher":"arXiv; DeepMind","event_date":"2022-04","available_at":"2022-04-29","explanation":"The preceding discussion identifies hallucinations and ungrounded guesses in Flamingo's open-ended dialogue. These establish limitations, but neither a comprehensive human comparison nor unanimity among careful observers."},{"id":"e2","relation":"context","excerpt":"truth-telling is not amongst them","locator":"Abstract","url":"https://link.springer.com/article/10.1007/s11023-022-09602-0","publisher":"Minds and Machines; Springer Nature","event_date":"2022-05-03","available_at":"2022-05-03","explanation":"The authors distinguish GPT-3's semantic competence from its unreliable truth-telling and challenge wholesale dismissals of its semantic abilities. This qualifies a strong reading of Marcus's dismissal without establishing broad human equivalence."}],"what_would_resolve":"Contemporaneous evaluations and documented observer judgments addressing broad human equivalence across the named systems under an explicit, defensible comparison.","retrospective":true,"plain_reading":true},"payload_hash":"4ef322adf71207f2e0d2c317cd75c5dd20f6c599acd992968d38c87acb7c889d","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T21:59:44Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-75a6f1-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-75a6f1-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-75a6f1-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T21:59:44Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-75a6f1-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T22:52:30Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-7679e7","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-7679e7-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-7679e7#assessment-marcus-the-new-science-of-alt-intelligence-7679e7-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-7679e7","subject_id":"gary-marcus","revision":2,"content_hash":"8f5e3cb7364d63fa8e91a61c6aad460c094a457e214ee55a1a402267385dfd66","normalized_proposition":"Success on a few problems with a particular strategy in no way guarantees that all problems can be solved the same way, and it is inductive folly to think otherwise, especially as the failure modes (unreliability, bizarre errors, failures of compositionality and discomprehension) have not changed since Fodor and Pinker pointed them out in 1988.","kind":"observation","target":"neural network AI","conditions":[],"original_horizon_text":null,"topic_tags":["reliability","scaling"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-7679e7-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"Success on a few problems with a particular strategy in no way guarantees that we can solve all problems in a similar way. It is sheer inductive folly to imagine otherwise, particular when the failure modes (unreliability, bizarre errors, failures in compositionality and discomprehension) have not changed since Fodor and Pinker pointed them out (in separate articles) in 1988.","locator":"paragraph 58","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-7679e7-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-7679e7","subject_id":"gary-marcus","normalized_proposition":"Success on a few problems with a particular strategy in no way guarantees that all problems can be solved the same way, and it is inductive folly to think otherwise, especially as the failure modes (unreliability, bizarre errors, failures of compositionality and discomprehension) have not changed since Fodor and Pinker pointed them out in 1988.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Success on a few problems with a particular strategy in no way guarantees that we can solve all problems in a similar way. It is sheer inductive folly to imagine otherwise, particular when the failure modes (unreliability, bizarre errors, failures in compositionality and discomprehension) have not changed since Fodor and Pinker pointed them out (in separate articles) in 1988.","locator":"paragraph 58","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":["Success on a few problems","in no way guarantees","all problems in a similar way","unreliability, bizarre errors, failures in compositionality and discomprehension","have not changed since Fodor and Pinker pointed them out","in separate articles","in 1988"],"our_interpretation":["The checkable historical assertion is that the broad kinds of failure identified in the 1988 critiques remained present in neural systems by May 14, 2022.","I read 'have not changed' as persistence of failure categories, rather than identical error rates, unchanged architectures, or no progress on individual problems.","I read compositionality as applying familiar components in unfamiliar combinations, and discomprehension as observable errors in interpreting meaning.","The target period follows the explicit comparison from 1988 through the post date. It does not extend to systems introduced after the post.","The statement about guarantees is a logical caution, and 'inductive folly' is an evaluative judgment. Neither is treated as a separate empirical forecast or as a prediction that neural methods can never succeed."],"support_condition":"The original critiques identify problems with systematic generalization and language processing, and research available by the post documents analogous compositional, comprehension, and unreliable or anomalous output failures in contemporary neural systems.","contradiction_condition":"The original articles do not identify the relevant problems, or evidence establishes that the broad failure categories had been overcome by the post date. Improvements on particular tasks alone do not contradict persistence of the categories."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Supported on the ordinary reading that broad failure categories persisted. The 1988 articles raise problems with systematic language processing and generalization. April 2022 research still documents compositional difficulties, semantic-understanding errors, hallucinations, and other unreliable outputs. The stronger reading that nothing relevant improved is not supported: PaLM's report records errors corrected by scaling, and Kirov and Cotterell report overcoming many specific Pinker–Prince objections. These improvements do not establish that the broad categories disappeared. This assessment does not endorse the rhetorical judgment 'inductive folly' or infer that neural approaches can never solve the remaining problems.","limitations":["The outcome depends on interpreting 'have not changed' as persistence of categories.","The cited tests cover particular systems and tasks, not every neural architecture.","The evidence does not establish a common cause for old and modern failures.","The historical observation ends on May 14, 2022; subsequent capabilities are outside its scope."],"evidence":[{"id":"e1","relation":"context","excerpt":"Several arguments for combinatorial structure in\nmental representations are then reviewed.","locator":"Abstract; Part III, Systematicity of cognitive representation","url":"https://www.arts.uwaterloo.ca/~celiasmi/courses/old_courses/Phil256/papers/fodor.pylyshyn.1988.Connectionism%20cog%20architec.bbs.pdf","publisher":"Cognition; full-text copy hosted by the University of Waterloo","event_date":"1988","available_at":"1988-12-31","explanation":"Fodor and Pylyshyn connect systematic understanding of related sentences to structured representations and challenge connectionist accounts on that basis. This establishes the historical comparison, not an empirical verdict on 2022 systems."},{"id":"e2","relation":"context","excerpt":"for certain words.","locator":"Introductory list of shortcomings of the Rumelhart–McClelland model","url":"https://blutner.de/NeuralNets/Texts/Pinker_Prince.htm","publisher":"Cognition; full-text reproduction hosted by Reinhard Blutner","event_date":"1988","available_at":"1988-12-31","explanation":"Pinker and Prince document failures in a neural language-learning model, including missing outputs and faulty linguistic generalizations. Their task was past-tense learning, so connecting these failures to modern comprehension errors is a comparison of broad categories."},{"id":"e3","relation":"supports","excerpt":"the tasks remain challenging.","locator":"Abstract, final sentence","url":"https://arxiv.org/abs/2204.03758v1","publisher":"arXiv; Kensen Shi and colleagues","event_date":"2022-04","available_at":"2022-04-07","explanation":"Tests involving SCAN and RobustFill found continuing difficulty combining learned components in new ways, even with modified Transformers. This supports persistence of compositional generalization problems while explicitly acknowledging progress."},{"id":"e4","relation":"supports","excerpt":"hallucinations, repetitive outputs","locator":"Section 6.3, Figure 9 and accompanying error analysis, page 20","url":"https://arxiv.org/pdf/2204.02311v1","publisher":"arXiv; Aakanksha Chowdhery and colleagues, Google","event_date":"2022-04","available_at":"2022-04-05","explanation":"The PaLM analysis identifies semantic-understanding errors, omitted reasoning steps, and anomalous outputs; Figure 9 shows that scaling corrected some but not all errors in each category. This supports persistent comprehension and reliability problems, while also providing evidence against unchanged error frequency."},{"id":"e5","relation":"context","excerpt":"obviates many","locator":"Section 6, Summary of Resolved and Outstanding Criticisms","url":"https://arxiv.org/pdf/1807.04783v1","publisher":"arXiv; Christo Kirov and Ryan Cotterell","event_date":"2018","available_at":"2018-07-12","explanation":"This direct revisit reports substantial improvements on the original past-tense problem and resolution of many specific objections. It challenges a literal claim of unchanged deficiencies, but does not establish elimination of the broad failure categories across neural AI."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"21a556adde5bdef158b31d0ce40cee2adcaec9bcd294a75be9b00356166185ac","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T21:59:44Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-75a6f1-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-75a6f1-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-75a6f1-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"peer_reviewed","audited":null,"derived":null,"metric":null},{"id":"e3","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e4","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e5","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T21:59:44Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-7679e7-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T22:52:30Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-854ec6","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-854ec6-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-854ec6#assessment-marcus-the-new-science-of-alt-intelligence-854ec6-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-854ec6","subject_id":"gary-marcus","revision":2,"content_hash":"dff38d1dee454a75df072ddde1212ebe57e2bb6e12f545bdd1414cadc0570334","normalized_proposition":"Gato may well be better than anything else we currently have, but the fact that it still doesn't really work, even after immense investment, should give us pause.","kind":"normative","target":"Gato / Alt Intelligence approach","conditions":[{"text":"may well be better than anything else we currently have","type":"qualifier","from_source":true},{"text":"even after all the immense investments","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["scaling","business-economics"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-854ec6-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"It may well be better than anything else we currently have, but the fact that it still doesn’t really work, even after all the immense investments that have been made in it, should give us pause.","locator":"paragraph 70","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-854ec6-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-854ec6","subject_id":"gary-marcus","normalized_proposition":"Gato may well be better than anything else we currently have, but the fact that it still doesn't really work, even after immense investment, should give us pause.","kind":"normative","conditions":["may well be better than anything else we currently have","even after all the immense investments"],"original_horizon_text":null,"statement":{"excerpt":"It may well be better than anything else we currently have, but the fact that it still doesn’t really work, even after all the immense investments that have been made in it, should give us pause.","locator":"paragraph 70","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":["may well be better than anything else we currently have","even after all the immense investments"],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"7320403da910a6c7adcfc351bfb7155d9ba037c22e21d514eb2752044a41bd7e","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T03:08:15Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-screen-anthropic-3"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-screen-openai-3"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T03:08:15Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-854ec6-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-9051c6","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-9051c6-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-9051c6#assessment-marcus-the-new-science-of-alt-intelligence-9051c6-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-9051c6","subject_id":"gary-marcus","revision":1,"content_hash":"1f4b2b88cf47d5a33830c2f48e7e5aa1b25c84a83fddae63a6714f1dde7a1917","normalized_proposition":"Virtually all of the world's software still depends on knowledge engineering.","kind":"observation","target":"world's software","conditions":[],"original_horizon_text":null,"topic_tags":["other"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-9051c6-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"Virtually all of the world’s software, from web browsers to spreadsheets to word processors, still depends on knowledge engineering, and Sutton sells that short.","locator":"paragraph 56","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-9051c6-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-9051c6","subject_id":"gary-marcus","normalized_proposition":"Virtually all of the world's software still depends on knowledge engineering.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Virtually all of the world’s software, from web browsers to spreadsheets to word processors, still depends on knowledge engineering, and Sutton sells that short.","locator":"paragraph 56","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":["Virtually all of the world’s software","from web browsers to spreadsheets to word processors","still depends on knowledge engineering"],"our_interpretation":["The examples suggest an ordinary reading of knowledge engineering as deliberately programmed rules, algorithms and domain knowledge. This is broader than the specialist meaning of building knowledge bases for expert systems.","Depends on means material functional dependence; it does not require every component to be handwritten.","For this ongoing state without a reporting cadence, the target is May 2022 up to publication. Earlier releases are background context.","Virtually all means nearly universal dependence across software categories, without imposing an invented percentage."],"support_condition":"Representative architecture or implementation evidence across the software population establishes nearly universal material dependence on explicitly engineered logic.","contradiction_condition":"Representative evidence establishes a substantial population of software whose functioning does not materially depend on such logic."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The primary sources are consistent with Marcus's examples: contemporary browsers and office software depend on engineered components. They do not establish the worldwide proportion. Calling all explicit programming knowledge engineering also makes a substantial interpretive choice. The evidence supports the examples without settling the full claim.","limitations":["The reviewed implementations are illustrative rather than a representative sample.","The meaning of knowledge engineering materially affects the assessment."],"evidence":[{"id":"e1","relation":"context","excerpt":"enforcing security characteristics.","locator":"Process mitigation policies","url":"https://chromium.googlesource.com/chromium/src/+/101.0.4951.64/docs/design/sandbox.md","publisher":"Chromium Project","event_date":"2022-05-09","available_at":"2022-05-09","explanation":"This documents material dependence on explicitly implemented policies in a browser. It supports Marcus's example under the broad reading, but does not measure the worldwide software population."},{"id":"e2","relation":"context","excerpt":"document conversion libraries","locator":"Final paragraph before sharing links","url":"https://blog.documentfoundation.org/blog/2022/05/05/libreoffice-733-community","publisher":"The Document Foundation","event_date":"2022-05-04","available_at":"2022-05-05","explanation":"The release announcement identifies an office suite's dependence on conversion libraries. Treating this as knowledge engineering is the evaluator's broad reading, and the example does not establish near-universal prevalence."}],"what_would_resolve":"A representative account of software architectures in May 2022, assessed under a clearly stated meaning of knowledge engineering.","retrospective":true,"plain_reading":true},"payload_hash":"b2dfe634abf69b8cd7460cb778c9eed1f8f85a97bb95e2e0a4c021677ec3ea3d","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T21:59:44Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-75a6f1-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-75a6f1-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-75a6f1-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"company_disclosure","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T21:59:44Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-9051c6-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T22:52:30Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-99f3a2","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-99f3a2-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-99f3a2#assessment-marcus-the-new-science-of-alt-intelligence-99f3a2-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-99f3a2","subject_id":"gary-marcus","revision":1,"content_hash":"64e44b772d771c2877197a3c8a015aeb6f2a7ccded2c43142168567897bea914","normalized_proposition":"So far, in most but not all domains, Big Data approaches have beaten careful knowledge engineering.","kind":"observation","target":"Big Data approaches","conditions":[{"text":"in most domains, not all","type":"qualifier","from_source":true},{"text":"for now","type":"qualifier","from_source":true}],"original_horizon_text":null,"topic_tags":["scaling"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-99f3a2-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"So far, in most domains (not all, to be sure) Big Data has triumphed (for now) over careful knowledge engineering.","locator":"paragraph 55","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-99f3a2-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-99f3a2","subject_id":"gary-marcus","normalized_proposition":"So far, in most but not all domains, Big Data approaches have beaten careful knowledge engineering.","kind":"observation","conditions":["in most domains, not all","for now"],"original_horizon_text":null,"statement":{"excerpt":"So far, in most domains (not all, to be sure) Big Data has triumphed (for now) over careful knowledge engineering.","locator":"paragraph 55","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":["So far","in most domains","not all","Big Data has triumphed","for now","over careful knowledge engineering"],"our_interpretation":["The surrounding discussion concerns AI application domains, rather than all software or economic activity.","I read triumphed as superior demonstrated task performance, and most as a majority of major AI domains.","Big Data includes learning from large collected or generated datasets; knowledge engineering means substantial hand-designed domain knowledge.","So far describes cumulative progress through publication. No historical start date is supplied.","Not all and for now restrict scope and time; they are not forecast antecedents."],"support_condition":"A broad, defensible comparison across major AI domains shows that data-driven methods outperform knowledge-engineered alternatives in a majority, while allowing exceptions.","contradiction_condition":"A comparably broad domain comparison shows that data-driven methods do not lead in a majority. A few exceptions cannot contradict a claim that explicitly allows them."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The record contains concrete successes for data-driven learning, including AlphaZero's results against Stockfish. Those examples fit Marcus's general direction but do not establish most domains. A majority verdict would depend on an evaluator-selected domain list and classification of hybrid systems. The available evidence therefore does not settle the claim.","limitations":["This is not a finding that knowledge engineering generally performed better.","The claim expressly permits exceptions.","Benchmark progress, deployment prevalence, and commercial success are different measures."],"evidence":[{"id":"e1","relation":"context","excerpt":"defeated Stockfish","locator":"Results discussion accompanying Table 2","url":"https://arxiv.org/html/1712.01815v1","publisher":"arXiv; DeepMind researchers","event_date":"2017","available_at":"2017-12-05","explanation":"This is a concrete success for learning from self-play against a program using extensive hand-engineered domain knowledge. It establishes a selected-domain example, not the claimed majority."},{"id":"e2","relation":"context","excerpt":"data related to artificial intelligence","locator":"Abstract","url":"https://arxiv.org/abs/2205.03468","publisher":"arXiv; AI Index authors","event_date":"2022","available_at":"2022-05-02","explanation":"This contemporary survey is relevant to checking breadth across AI. Its readable abstract does not enumerate domains won by data-driven systems against knowledge-engineered alternatives, so it cannot settle the majority assertion."}],"what_would_resolve":"A justified cross-domain inventory of results through May 14, 2022, with consistent domain boundaries, identified knowledge-engineered comparators, and explicit treatment of hybrid methods.","retrospective":true,"plain_reading":true},"payload_hash":"f5fa525e6d059ef48f98334cccafec77289ceb8f03596a9c3fc6a8c85388e02c","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T21:59:44Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-75a6f1-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-75a6f1-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-75a6f1-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"secondary_summary","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T21:59:44Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-99f3a2-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T22:52:30Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-9e7096","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-9e7096-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-9e7096#assessment-marcus-the-new-science-of-alt-intelligence-9e7096-a2"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-9e7096","subject_id":"gary-marcus","revision":1,"content_hash":"ee2b37c26bc69269d9691f68e54d75ed2eb4f1d071af9ecfc8b361205a3502bf","normalized_proposition":"Gato does not work like the brain, learn like a child, understand language, or align with human values, and cannot be trusted with mission-critical tasks.","kind":"observation","target":"Gato","conditions":[],"original_horizon_text":null,"topic_tags":["capabilities","reliability","safety"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-9e7096-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"As I have written, I am fine with thinking of Gato as an “Alt Intelligence”— an interesting exploration in alternative ways to build intelligence—but we need to take it in context: it doesn’t work like the brain, it doesn’t learn like a child, it doesn’t understand language, it doesn’t align with human values, and it can’t be trusted with mission-critical tasks.","locator":"paragraph 69","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-9e7096-a2","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-9e7096","subject_id":"gary-marcus","normalized_proposition":"Gato does not work like the brain, learn like a child, understand language, or align with human values, and cannot be trusted with mission-critical tasks.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"As I have written, I am fine with thinking of Gato as an “Alt Intelligence”— an interesting exploration in alternative ways to build intelligence—but we need to take it in context: it doesn’t work like the brain, it doesn’t learn like a child, it doesn’t understand language, it doesn’t align with human values, and it can’t be trusted with mission-critical tasks.","locator":"paragraph 69","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":["Gato","doesn’t work like the brain","doesn’t learn like a child","doesn’t understand language","doesn’t align with human values","can’t be trusted with mission-critical tasks"],"our_interpretation":["Assess the original Gato described immediately before the post, not subsequent models.","Use May 12–14, 2022: the initial public report through the post date is the latest reportable state.","Read the brain and child comparisons as comparisons of operating and learning methods, not a denial of every biological inspiration.","Read language understanding as dependable comprehension beyond selected demonstrations; fluent output alone does not settle it.","Read alignment and mission-critical trust as dependable behavior consistent with intended human goals and adequate reliability where failure has serious consequences.","Support for the complete claim requires evidence for all five assertions. An untested capability is not automatically an absent capability."],"support_condition":"Architecture and learning evidence establish the asserted differences, and suitable language, alignment, and reliability evaluations substantiate the remaining categorical limitations.","contradiction_condition":"Direct evidence establishes a capability or correspondence that clearly conflicts with at least one assertion under the stated reading. Selected language examples alone do not establish dependable general comprehension."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The original Gato report, available May 12, 2022, describes a transformer trained offline with supervised learning and says harm-mitigation tools require further research before deployment [e6]. It also reports rudimentary dialogue, captioning, and instruction-following abilities [e6]. These demonstrations do not establish dependable language comprehension, and the deployment warnings do not establish categorical misalignment or failure against a specified mission-critical reliability requirement [e6]. The report therefore neither substantiates all five assertions nor supplies a clear contradiction under the stated reading [e6].","limitations":["The evidence is the developers' original report, rather than independent validation of all five assertions [e6].","The architecture and training descriptions do not settle every interpretation of the brain and child comparisons [e6].","The report supplies neither comprehensive comprehension and alignment tests nor an application-specific mission-critical reliability assessment; missing tests do not establish absent capabilities [e6]."],"evidence":[{"id":"e6","relation":"context","excerpt":"require further research","locator":"Section 7, Broader Impact; architecture in Sections 1–2 and language demonstrations in Section 4.3","url":"https://arxiv.org/html/2205.06175v1","publisher":"arXiv; DeepMind","event_date":"2022-05","available_at":"2022-05-12","explanation":"The report describes a transformer trained offline with supervised learning and cautions that deployment safeguards need further work. It also reports rudimentary dialogue, captioning, and instruction-following abilities, which qualify a literal denial of all language capability without establishing dependable comprehension or value alignment."}],"what_would_resolve":"Defined comparison criteria and evaluations of the original Gato as released in May 2022, covering operating and learning methods, dependable language comprehension, human-value alignment, and reliability in a specified mission-critical application. These could substantiate all five assertions or clearly contradict at least one.","retrospective":true,"plain_reading":true},"payload_hash":"9fe8591513374c52a9e7d1df2c4e7933bd4c54d9f5ba6394f1cb92ff59985bcc","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T12:42:08Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-11b850-2x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-11b850-2x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-11b850-2x-final-anthropic-2"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e6","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T12:42:08Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-9e7096-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-a01724","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-a01724-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-a01724#assessment-marcus-the-new-science-of-alt-intelligence-a01724-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-a01724","subject_id":"gary-marcus","revision":2,"content_hash":"d79b6f951d5fce891e26ffbad8479fbe774a2ca6af35f83d24ce77e15cad0bda","normalized_proposition":"It is sheer hubris to believe that AGI can be built without learning from how humans reason, understand the world, and represent and acquire language and concepts.","kind":"normative","target":"scaling-only view of AGI","conditions":[],"original_horizon_text":null,"topic_tags":["agi-timelines","scaling"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-a01724-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"It is sheer hubris to believe otherwise.","locator":"paragraph 73","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-a01724-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-a01724","subject_id":"gary-marcus","normalized_proposition":"It is sheer hubris to believe that AGI can be built without learning from how humans reason, understand the world, and represent and acquire language and concepts.","kind":"normative","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"It is sheer hubris to believe otherwise.","locator":"paragraph 73","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this a recommendation or a value judgment.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["value_judgment"]},"payload_hash":"f4ced30be66a6a1d5dc14fb24e42bce0d9af949564f40f280dc25f9fbedef853","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T03:08:15Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-screen-anthropic-3"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-screen-openai-3"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T03:08:15Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-a01724-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-ca7f59","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-ca7f59-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-ca7f59#assessment-marcus-the-new-science-of-alt-intelligence-ca7f59-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-ca7f59","subject_id":"gary-marcus","revision":1,"content_hash":"8895e7c53e7033143794f130fcbfe8b05683103f66ab5aea3980cf37e8856395","normalized_proposition":"DeepMind's Gato achieves novel cross-modal feats but remains unreliable, mixing brilliance with total discomprehension.","kind":"observation","target":"Gato","conditions":[],"original_horizon_text":null,"topic_tags":["reliability","capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-ca7f59-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"DeepMind’s newest star, just unveiled, Gato, is capable of cross-modal feats never seen before in AI, but still, when you look in the fine print, remains stuck in the same land of unreliability, moments of brilliance coupled with absolute discomprehension:","locator":"paragraph 32","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-ca7f59-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-ca7f59","subject_id":"gary-marcus","normalized_proposition":"DeepMind's Gato achieves novel cross-modal feats but remains unreliable, mixing brilliance with total discomprehension.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"DeepMind’s newest star, just unveiled, Gato, is capable of cross-modal feats never seen before in AI, but still, when you look in the fine print, remains stuck in the same land of unreliability, moments of brilliance coupled with absolute discomprehension:","locator":"paragraph 32","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":["Gato","just unveiled","cross-modal feats never seen before in AI","unreliability","moments of brilliance coupled with absolute discomprehension"],"our_interpretation":["Novelty concerns Gato's combined range of language, vision, game-playing and real robot behavior in one model with shared weights, rather than the first existence of multimodal AI.","Brilliance and discomprehension describe strong performances alongside basic observable failures, rather than measurable mental states.","Just unveiled identifies the launch version and its May 12–14, 2022 public record."],"support_condition":"The launch record documents a distinctive combination of modalities and tasks, substantial successful performance, and recurrent basic failures.","contradiction_condition":"Comparable earlier systems already demonstrate the same combined coverage, or launch evaluations rebut the alleged failures and establish reliable performance across the relevant tasks."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Gato's launch paper documents the combination Marcus describes: broad abilities in one model, substantial successes and frequent superficial or factually incorrect replies. Support depends on reading novelty as the combined coverage. Earlier multimodal systems already existed. The evidence supports observable unreliability, not a literal claim about an absence of understanding.","limitations":["Novelty is scoped to the combination of capabilities.","The results are developer-reported.","The qualitative dialogue evidence supplies no overall error rate or measure of comprehension."],"evidence":[{"id":"e1","relation":"supports","excerpt":"factually incorrect","locator":"Figure 7 caption, page 10; accompanying results in Sections 4 and 6","url":"https://arxiv.org/pdf/2205.06175v1","publisher":"arXiv; DeepMind","event_date":"2022-05-12","available_at":"2022-05-12","explanation":"The launch paper reports one model spanning dialogue, captions, games and real robot control, including human-level or better performance on 23 Atari games, while acknowledging frequent poor dialogue responses. Its prior-work comparison supports novelty in that combined coverage, rather than novelty of multimodality itself."},{"id":"e2","relation":"context","excerpt":"spanning multiple domains","locator":"Abstract, third sentence","url":"https://arxiv.org/abs/1706.05137","publisher":"arXiv; Kaiser and colleagues","event_date":"2017-06-16","available_at":"2017-06-16","explanation":"This earlier model combined speech, image and language tasks. It limits any suggestion that Gato invented multimodal models, but does not establish the same combined coverage of games and real robot control."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"642ea255be1886b15e9e100680c5b7ddecd4138f5142707593f1c48d700b6d4e","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T21:59:47Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-a08647-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-a08647-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-a08647-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T21:59:47Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-ca7f59-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T22:52:30Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-d2de9c","standing":"current","assessment_label":"Unresolved","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-d2de9c-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-d2de9c#assessment-marcus-the-new-science-of-alt-intelligence-d2de9c-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-d2de9c","subject_id":"gary-marcus","revision":1,"content_hash":"84f6b4b3b775560183a3be33f6ff3711242c8043bcdb32ae3dd819f77b5f19fd","normalized_proposition":"Scaling is currently the predominant strand of work in AI that substitutes data for intelligence.","kind":"observation","target":"scaling","conditions":[],"original_horizon_text":null,"topic_tags":["scaling"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-d2de9c-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"Right now, the predominant strand of work within Alt Intelligence is the idea of scaling.","locator":"paragraph 3","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-d2de9c-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-d2de9c","subject_id":"gary-marcus","normalized_proposition":"Scaling is currently the predominant strand of work in AI that substitutes data for intelligence.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"Right now, the predominant strand of work within Alt Intelligence is the idea of scaling.","locator":"paragraph 3","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":["Right now","the predominant strand of work within Alt Intelligence","the idea of scaling"],"our_interpretation":["Alt Intelligence means the data-driven approaches described in the surrounding passage, rather than all AI research.","Predominant means the leading research direction. Prominent examples alone do not establish this comparative claim.","For this ongoing research trend, I use the latest research cycle before publication: 2022 through May 14. Marcus supplied no reporting period or measure of predominance."],"support_condition":"Representative evidence about research programs, publications or investment establishes scaling as the leading direction within the described approaches.","contradiction_condition":"Representative evidence establishes that another direction predominated within those approaches."},"axis":"historical_observation","outcome":"unresolved","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Contemporaneous sources show that scaling was an important research direction. They do not establish the stronger claim that it predominated within the approaches Marcus describes. Treating a selection of prominent models as a representative comparison would overstate the evidence.","limitations":["The category and measure of predominance are unspecified.","The retrieved evidence measures compute trends and individual programs, rather than comparative research prevalence."],"evidence":[{"id":"e1","relation":"context","excerpt":"fast-growing compute requirements","locator":"Abstract, final sentence","url":"https://arxiv.org/abs/2202.05924v1","publisher":"arXiv; Sevilla and colleagues","event_date":"2022-02","available_at":"2022-02-11","explanation":"The study documents increasing training compute in advanced systems. It does not establish scaling's share of the research population Marcus describes."},{"id":"e2","relation":"context","excerpt":"unlocking new capabilities","locator":"Introduction, sentence immediately before the training section","url":"https://research.google/blog/pathways-language-model-palm-scaling-to-540-billion-parameters-for-breakthrough-performance","publisher":"Google Research","event_date":"2022-04-04","available_at":"2022-04-04","explanation":"Google presented PaLM as a major scaling effort and attributed capability gains to scale. One prominent program cannot establish predominance across a research field."}],"what_would_resolve":"A representative comparison of research directions through May 14, 2022, using a justified definition of the approaches Marcus describes.","retrospective":true,"plain_reading":true},"payload_hash":"f1c0b06c73bf3600f3eba8222b4d5d71fff0056a6af752e664c839c9f4061f08","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T21:59:47Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-a08647-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-a08647-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-a08647-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"company_disclosure","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T21:59:47Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-d2de9c-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T22:52:30Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-d724f8","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-d724f8-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-d724f8#assessment-marcus-the-new-science-of-alt-intelligence-d724f8-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-d724f8","subject_id":"gary-marcus","revision":1,"content_hash":"e899d54251b9468c66bf6508b5598fb3d6cebd18d851006957cce90c8db8224c","normalized_proposition":"DALL-E 2 is fairly prone to uncomprehending errors such as failing to render 'a red cube on top of a blue cube'.","kind":"observation","target":"DALL-E 2","conditions":[],"original_horizon_text":null,"topic_tags":["reliability"],"concerns_providers":["openai"],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-d724f8-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"But it’s also pretty prone to surprising stumbles, like this uncomprehending reply to “a red cube on top of a blue cube”.","locator":"paragraph 19","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-d724f8-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-d724f8","subject_id":"gary-marcus","normalized_proposition":"DALL-E 2 is fairly prone to uncomprehending errors such as failing to render 'a red cube on top of a blue cube'.","kind":"observation","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"But it’s also pretty prone to surprising stumbles, like this uncomprehending reply to “a red cube on top of a blue cube”.","locator":"paragraph 19","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":["pretty prone to surprising stumbles","a red cube on top of a blue cube","DALL-E 2"],"our_interpretation":["The ordinary reading is that the original DALL-E 2 repeatedly mishandles simple compositional instructions, including assigning the requested colors to objects.","Uncomprehending describes the observable mismatch between request and output; it does not establish an internal mental state.","Pretty prone supplies no numerical failure threshold and is assessed as a recurring failure mode.","For this ongoing observation, I use the latest public evaluation cycle, April 1 through May 14, 2022, surrounding the original model's introduction."],"support_condition":"Original-model evaluations document recurring compositional or attribute-binding failures, including the cube example or an equivalent test.","contradiction_condition":"Representative original-model tests establish that these failures are rare and show that the cited example was misidentified or did not reflect the model's behavior."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The original DALL-E 2 paper independently documents the exact cube example and identifies recurring difficulty assigning attributes to the correct objects. That supports the ordinary qualitative reading of Marcus's claim. It does not establish a numerical error rate or prove anything about the model's internal understanding.","limitations":["Support concerns a recurring compositional failure mode, not a measured failure rate across all prompts.","The evidence is the developers' technical evaluation."],"evidence":[{"id":"e1","relation":"supports","excerpt":"binding attributes to objects","locator":"Section 7, Limitations and Risks; Figures 14 and 15","url":"https://arxiv.org/html/2204.06125v1","publisher":"arXiv; OpenAI","event_date":"2022-04","available_at":"2022-04-13","explanation":"The authors discuss the exact red-cube/blue-cube prompt in Figure 14 and report recurring object–attribute mix-ups in reconstructions in Figure 15. This supports a recurring compositional weakness in the original model, without establishing a population-wide error rate."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"a644ab9d9844650d5ab5cbb4e766d37337a8409534cc1b76099d1b1e042e7246","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T21:59:47Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-a08647-4x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-a08647-4x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-a08647-4x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T21:59:47Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-d724f8-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T22:52:30Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-f493ea","standing":"current","assessment_label":"Can’t be checked","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-f493ea-a1","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-f493ea#assessment-marcus-the-new-science-of-alt-intelligence-f493ea-a1"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-f493ea","subject_id":"gary-marcus","revision":1,"content_hash":"00bc6f4726dcfca7d29f059ec0fdc311f92e79eb8c4ad3bccd6bc9dffa394ff1","normalized_proposition":"The persistent problem of discomprehension may signal that AI needs a fundamental rethink.","kind":"normative","target":"AI field","conditions":[],"original_horizon_text":null,"topic_tags":["scaling","capabilities"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-f493ea-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"To some (such as myself), the abiding problem of discomprehension might—despite the immense progress—signal the need for a fundamental rethink, such as the one that Davis and I offered in our book Rebooting AI.","locator":"paragraph 38","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-f493ea-a1","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-f493ea","subject_id":"gary-marcus","normalized_proposition":"The persistent problem of discomprehension may signal that AI needs a fundamental rethink.","kind":"normative","conditions":[],"original_horizon_text":null,"statement":{"excerpt":"To some (such as myself), the abiding problem of discomprehension might—despite the immense progress—signal the need for a fundamental rethink, such as the one that Davis and I offered in our book Rebooting AI.","locator":"paragraph 38","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":[],"our_interpretation":[],"support_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim.","contradiction_condition":"Not checked against evidence: both screens found that public evidence could not settle this claim."},"axis":"not_applicable","outcome":"not_adjudicable","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"Not checked against evidence. Two independent screens, one from each provider, both found this stated as a chance or a possibility, or at medium or low confidence.","limitations":[],"evidence":[],"what_would_resolve":null,"retrospective":true,"uncheckable":["probability_language"]},"payload_hash":"1c4e3fe4bd9fd836e33ca98932cbf2a9f3312c104a42fc92afc8e63db0d4e3f3","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T02:32:05Z","drafter":{"provider":"anthropic","model":"claude-opus-5-5"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-screen-anthropic-2"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-screen-openai-2"}],"final_text":null,"status":"current","notice":null,"superseded_by":null,"evidence_meta":[],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T02:32:05Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-f493ea-assessment-published-a1"],"last_substantive_change_at":"2026-10-29T17:00:18Z"}},{"url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-fdddcc","standing":"current","assessment_label":"Supported","versions":[{"assessment_id":"marcus-the-new-science-of-alt-intelligence-fdddcc-a2","status":"current","url":"https://punditclaims.ai/claims/marcus-the-new-science-of-alt-intelligence-fdddcc#assessment-marcus-the-new-science-of-alt-intelligence-fdddcc-a2"}],"claim":{"claim_id":"marcus-the-new-science-of-alt-intelligence-fdddcc","subject_id":"gary-marcus","revision":2,"content_hash":"1ad42e5adac83b168af4ca71e36a1278dc6d85cf9ab56c95d09d7ab6ad559fa2","normalized_proposition":"Many neural net enthusiasts have argued against symbols for decades, and symbolic descriptions like 'red cube on blue cube' still elude the state of the art in 2022.","kind":"observation","target":"state-of-the-art AI","conditions":[],"original_horizon_text":"in 2022","topic_tags":["capabilities","reliability"],"concerns_providers":[],"selection_stratum":"declared-lead","occurrences":[{"id":"marcus-the-new-science-of-alt-intelligence-fdddcc-o1","statement_date":"2022-05-14","statement_date_precision":"day","excerpt":"The history missed is the the fact that many neural net enthusiasts, have argued against symbols for decades, and the reality that is missed is the above-documented fact that symbolic descriptions like “red cube on blue cube” still elude the state-of-the-art in 2022.)","locator":"paragraph 48","document":{"url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","title":"The New Science of Alt Intelligence","publisher":"Marcus on AI","published_at":"2022-05-14T17:02:35Z","date_uncertain":false,"date_provenance":"JSON-LD datePublished"},"observed_at":"2026-10-29T00:05:24Z","observed_via":"live","archive_url":null,"access_state":"full_public","snapshot_hash":"b5456ff99a853d55856ae13c7d7571ff3970a73d8fefc55e7d34fd90527b6f81","receipts":1}],"first_statement_date":"2022-05-14","related":[],"current":{"assessment_id":"marcus-the-new-science-of-alt-intelligence-fdddcc-a2","payload":{"claim_id":"marcus-the-new-science-of-alt-intelligence-fdddcc","subject_id":"gary-marcus","normalized_proposition":"Many neural net enthusiasts have argued against symbols for decades, and symbolic descriptions like 'red cube on blue cube' still elude the state of the art in 2022.","kind":"observation","conditions":[],"original_horizon_text":"in 2022","statement":{"excerpt":"The history missed is the the fact that many neural net enthusiasts, have argued against symbols for decades, and the reality that is missed is the above-documented fact that symbolic descriptions like “red cube on blue cube” still elude the state-of-the-art in 2022.)","locator":"paragraph 48","url":"https://garymarcus.substack.com/p/the-new-science-of-alt-intelligence","date":"2022-05-14","document_title":"The New Science of Alt Intelligence","observed_at":"2026-10-29T00:05:24Z"},"criteria":{"from_speaker":["many neural net enthusiasts","argued against symbols for decades","symbolic descriptions like “red cube on blue cube”","still elude the state-of-the-art in 2022"],"our_interpretation":["The historical clause concerns a substantial strand of connectionist opposition to symbolic accounts of cognition; it does not claim unanimity among neural-network researchers.","Decades means that this position is documented over a period spanning at least twenty years, without requiring the same individuals to maintain it throughout.","In the surrounding discussion of unreliability, elude means that leading models could not reliably satisfy such compositional descriptions, not that no successful image could ever be generated.","The target period includes the unspecified earlier historical record through publication. The contemporary-performance clause is restricted to 2022 through May 14."],"support_condition":"Historical records establish a long-standing, substantial connectionist position opposing symbolic processing, and original contemporary evidence documents unreliable compositional rendering by a leading 2022 image model.","contradiction_condition":"The historical record fails to contain the asserted substantial opposition, or contemporary evidence establishes reliable handling of the described compositional image tasks. Hybrid research and isolated successful images do not alone contradict the claim."},"axis":"historical_observation","outcome":"supported","horizon_state":"not_applicable","evaluation_as_of":"2026-10-28","rationale":"The 1997 and 2019 Stanford Encyclopedia of Philosophy accounts describe many connectionists as rejecting symbolic explanations of cognition, documenting that strand across more than twenty years [e1] [e2]. Both accounts also recognize connectionists who accommodate symbolic processing [e1] [e2]. The DALL·E 2 paper released on April 13, 2022 reports difficulty binding colors to objects in the prompt 'a red cube on top of a blue cube' [e3]. Together, these sources support the historical claim and the contemporary capability claim when 'elude' means unreliable rendering, rather than complete inability [e1] [e2] [e3].","limitations":["The historical evidence comprises two versions of one scholarly account, not independent surveys or a count of advocates [e1] [e2].","The accounts do not establish unanimous opposition or that the same individuals maintained their positions throughout the period [e1] [e2].","The model evidence concerns DALL·E 2 and a GLIDE comparison, not an exhaustive evaluation of every system available by May 14, 2022 [e3].","GLIDE's relative advantage does not establish reliable handling of the task, and the reported difficulties do not establish complete inability [e3]."],"evidence":[{"id":"e1","relation":"supports","excerpt":"eliminate symbolic\nprocessing","locator":"The Shape of the Controversy between Connectionists and Classicists","url":"https://plato.stanford.edu/archives/fall1998/entries/connectionism","publisher":"Stanford Encyclopedia of Philosophy","event_date":"1997","available_at":"1997-07-29","explanation":"This dated scholarly account documents an explicit anti-symbolic strand well before the post. It is historical synthesis rather than a primary statement by an advocate."},{"id":"e2","relation":"supports","excerpt":"many connectionists","locator":"Section 5, The Shape of the Controversy between Connectionists and Classicists","url":"https://plato.stanford.edu/entries/connectionism","publisher":"Stanford Encyclopedia of Philosophy","event_date":"2019","available_at":"2019-08-16","explanation":"The revised account still describes substantial opposition to explaining cognition through symbolic processing, alongside accommodation and hybrid approaches. Together with the 1997 version, it supports the ordinary historical reading across decades; these are versions of one account, not independent surveys."},{"id":"e3","relation":"supports","excerpt":"binding attributes to objects","locator":"Section 7, Limitations and Risks; Figure 14","url":"https://arxiv.org/html/2204.06125v1","publisher":"arXiv; OpenAI researchers","event_date":"2022-04","available_at":"2022-04-13","explanation":"The surrounding passage and Figure 14 examine the red-cube-on-blue-cube prompt and report difficulty assigning the requested colors to the separate objects. This directly supports unreliability on Marcus's example class."}],"what_would_resolve":null,"retrospective":true,"plain_reading":true},"payload_hash":"2c1caf6ea1e9bbca974ce9baa0607d9fff7d1d40fd654296166c7a5b87e6040a","policy_version":"2026-10-28.7","recorded_at":"2026-10-29T22:11:22Z","drafter":{"provider":"openai","model":"gpt-6-astra"},"reviewers":[{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-fdddcc-1x-review-anthropic-1"},{"provider":"openai","model":"gpt-6-astra","run_id":"marcus-the-new-science-of-alt-intelligence-fdddcc-1x-review-openai-1"}],"final_text":{"provider":"anthropic","model":"claude-opus-5-5","run_id":"marcus-the-new-science-of-alt-intelligence-421790-3x-final-anthropic-1"},"status":"current","notice":null,"superseded_by":null,"evidence_meta":[{"id":"e1","source_type":"secondary_summary","audited":null,"derived":null,"metric":null},{"id":"e2","source_type":"secondary_summary","audited":null,"derived":null,"metric":null},{"id":"e3","source_type":"preprint","audited":null,"derived":null,"metric":null}],"public_from":null,"public_until":null},"history":[],"review":{"review_due":false,"triggers":[],"last_completed_review_at":"2026-10-29T22:11:22Z","next_review_due_at":null,"horizon_now":"not_applicable"},"event_ids":["marcus-the-new-science-of-alt-intelligence-fdddcc-assessment-published-a2"],"last_substantive_change_at":"2026-10-29T22:52:30Z"}}]}