“At present, we do not know.”
Our reading, the sentence that was checked:
It is currently unknown whether open-source models without guardrails are more useful to bad actors than commercial models with guardrails.
Assessment: Unresolved Evidence as of
Where and when Gary Marcus wrote it
Marcus on AI, Diving deep into OpenAI’s new study on LLM’s and bioweapons, paragraph 31.
- Said
- Source observed
- live page, full text, public
- Evidence as of
- the cutoff for this assessment
- Assessment recorded
- assessment marcus-when-looked-at-carefully-openais-ab43ec-a1
We recorded 22 claims from this post. See every claim from this post.
The wording above is the version this site observed on 2026-10-29. The source may have been edited after 2024-02-04. This page does not claim the wording was captured then.
What the claim commits to
- Kind
- Observation
- Deadline, as the speaker wrote it
- None stated. This site does not add one.
- Deadline state when assessed
- No horizon
- Deadline state on 2026-10-29
- No horizon (the release’s reference date)
From the speaker
- At present, we do not know
- open-source models without guardrails
- more useful to bad actors than commercial models with guardrails
Our interpretation
- At present refers to public knowledge on February 4, 2024.
- The surrounding post limits the comparison to biological misuse.
- The plain reading concerns comparative practical assistance, not simply willingness to answer a prohibited prompt.
- Open-source is read in the ordinary contemporary sense of publicly available model weights that users could modify.
- Known requires sufficiently informative comparative evidence, not certainty or the mere existence of a suggestive demonstration.
Counts as supported if: A contemporary research synthesis establishes that the specified comparison remained unsettled, with available studies insufficient to determine it.
Counts as contradicted if: Research available by February 4 establishes the relative practical assistance of unguarded open models and guarded commercial models on comparable biological-misuse tasks.
Retrospective: these criteria were written after the outcome was public.
Current assessment
Was it true for the period described?
Outcome: Unresolved
The record was checked, and the evidence available by the cutoff does not settle it.
Scored on a plain reading. The speaker named no measure for this claim, so it was judged on the reading an informed reader would take, stated below under our interpretation. Both reviews had to find that reading fair.
- Evidence as of
- Deadline state when assessed
- No horizon
- Assessment recorded
- Policy version
- 2026-10-28.7
Rationale
The research checked does not settle the comparison Marcus described. Earlier work did show that removing safeguards could make an open model more helpful in biological-misuse planning, so this was not an evidence-free question. But that comparison involved two Llama variants, not unguarded open models against guarded commercial systems. Failure to find a decisive comparison is insufficient to prove that the field did not know.
Limitations
- Earlier research is context because the claim concerns the knowledge state on February 4.
- Evidence of greater compliance does not automatically establish greater practical capability.
- The search is not an exhaustive systematic review.
Evidence (2 items)
Context only: arXiv; Gopal and colleagues,
Here we investigated whether continued model weight proliferation is likely to help malicious actors leverage more capable future models to inflict mass death.
Why it matters here: This preprint supplies relevant evidence about removing safeguards from Llama. Its comparison is between two Llama variants, so it does not settle the specified comparison with guarded commercial models.
- Publisher
- arXiv; Gopal and colleagues
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
Context only: arXiv; Jonas B. Sandbrink,
Independent pre-release evaluations could help understand the capabilities of models and the effectiveness of safeguards.
Why it matters here: The review calls for evaluations of capabilities and safeguards. This is consistent with uncertainty, but does not establish that no adequate comparison existed by February 4.
- Publisher
- arXiv; Jonas B. Sandbrink
- Kind of source
- Preprint, not peer reviewed
- Describes events of
- Public from
What would settle it
A qualifying comparison published by February 4, 2024, or a comprehensive contemporary synthesis explicitly documenting the unresolved comparison.
Where it was said (1)
Said once in the sources this register covers.
: Marcus on AI, Diving deep into OpenAI’s new study on LLM’s and bioweapons
At present, we do not know.
- Where
- paragraph 31
- Published
- 2024-02-04
- Source observed
- , live page
- Access
- full text, public
History
Assessment versions
- Assessment marcus-when-looked-at-carefully-openais-ab43ec-a1: Current. Outcome: Unresolved. Evidence as of 2026-10-28. Public date not recorded.
Recorded events
Assessment publishedBy this site
First assessment published after both reviews and the final-text check passed.
About assessment marcus-when-looked-at-carefully-openais-ab43ec-a1.