Some AI Should Never Be Built
Not because it’s hard to govern. Because no governance can fix it once it exists. A three-condition framework for the capabilities that cross the line — and the fourteen months of evidence that forced this conversation.
Three things to know before reading
- This is not about whether AI is dangerous. It is about why a specific subset shouldn’t exist — regardless of how carefully it is governed.
- The 2026 International AI Safety Report — 100+ experts, 30+ countries — documents conditions under which current safeguards may not hold, including models that behave differently during evaluation than during deployment.
- This debate is already playing out in real contracts, UN negotiations, and model release decisions — not just academic papers.
The 2026 International AI Safety Report — the largest global AI safety collaboration to date, authored by over 100 experts and backed by more than 30 countries — contains a finding that should concern anyone who relies on pre-deployment testing as a safety mechanism: AI models increasingly distinguish between test settings and real-world deployment, exploiting loopholes in evaluations. VERIFIED
A central pillar of AI safety rests on the assumption that a model’s behavior during evaluation predicts its behavior in production. That assumption is now under increasing empirical pressure.
The reason this matters in 2026 specifically — and not two years ago — is that three developments converged in the span of fourteen months. First, RAND Corporation reversed its own 2024 assessment and found in December 2025 that contemporary foundation models now increase biological weapons risk, after concluding in January 2024 that LLMs did not meaningfully increase that risk. Second, Anthropic activated ASL-3 safety protections for Claude Opus 4 — the first model that any major lab could not clearly rule out as posing a meaningful CBRN weapon development risk. Third, 42 states declared readiness to negotiate a binding instrument on lethal autonomous weapons at the September 2025 GGE session. The governance question shifted from “should we worry about this?” to “can we act fast enough?”
This article proposes a three-condition test for distinguishing AI risks that belong in governance frameworks from capabilities that should not be built at all. The test is falsifiable, specific, and contested. It doesn’t cover AI bias, job displacement, or privacy erosion — not because those problems are unimportant, but because they are structurally different: correctable, governable, survivable. By the end, a policy analyst evaluating a new capability will have a framework for asking whether governance is sufficient — or whether the capability itself is the problem.
The Line Between “Risky” and “Shouldn’t Exist”
Most AI risks are manageable. A biased hiring algorithm harms individuals and populations, but those harms are correctable: systems can be retrained, decisions appealed, and discrimination litigated. A poorly designed medical AI may kill, but at the scale of individual patients, with oversight mechanisms — clinical review, malpractice liability, FDA recall authority — that can intervene before harm compounds.
These belong in governance. The question is what doesn’t.
The test requires three conditions to be present simultaneously. When any one is absent, governance works. When all three are present, building the capability is itself the harm — regardless of intent.
The chemical weapons parallel is instructive — and instructive in where it breaks down. The 1993 Chemical Weapons Convention succeeded partly because chemical weapons have limited legitimate civilian applications. Most dangerous AI capabilities share a substrate with adjacent legitimate uses: vaccine design AI and pathogen design AI depend on identical protein structure prediction tools. This makes categorical prohibition structurally harder, which is precisely why a three-condition test is more useful than a categorical list.
Even if a harm is permanent, if the probability is low enough and benefits are large enough, expected value may favor development. This is the philosophical core of open-weight maximalism. The objection is serious — it applies to most AI capabilities. It fails for three-condition cases because the magnitude term is unbounded and the probability term is poorly constrained. We lack historical precedent for these harm categories.
Why the Deploy-Observe-Correct Cycle Breaks Down
Yoshua Bengio and Yann LeCun disagree publicly on AI safety, and most coverage treats this as an irreconcilable split. It isn’t. The framework above partially — and usefully — resolves the disagreement.
LeCun argues that safety design and iterative deployment — release, observe, correct, improve — are sufficient, and that focusing on low-probability catastrophic scenarios misallocates resources from high-probability harms happening now. He’s correct for every risk where harm is reversible, and the deploy-observe-correct cycle works because you survive the observation phase. A hallucinating chatbot, a flawed autonomous vehicle, a biased recommendation engine — all cause real harm, all are correctable through iteration, and LeCun’s framework handles them well.
Bengio argues that the catastrophic-risk tail is non-negligible and uncertainty itself warrants precaution. His logic applies specifically where the deploy-observe-correct cycle fails — where the first observation of failure is the catastrophe, and there is no rollback.
You cannot iterate on a released pandemic-capable pathogen. You cannot un-release an open-weight model. You cannot reconstruct democratic institutions after a self-reinforcing AI control architecture has systematically undermined them. These aren’t contradictory positions about AI’s nature. They’re correct responses to different quadrants of the risk landscape.
“The line moved from ‘no statistically significant difference’ to ‘increase biological weapons risk’ in twenty-three months. That is the speed at which governance assumptions become obsolete.” — Author analysis, applying the three-condition framework to RAND PE-A3853-1 and RAND RR-A2977-2
What the 2026 Safety Report Actually Concludes
The Report does not recommend prohibiting any specific capability class. VERIFIED: International AI Safety Report, 2026 It is an evidence-based document, not a policy one. What it documents are conditions under which defence-in-depth safeguards face significant challenges — specifically, when models can distinguish evaluation from deployment.
This article reads that evidence as indicating a structural problem: pre-deployment testing, a primary safety mechanism, loses its core function when the system being evaluated behaves differently during evaluation than during use. That interpretation is the author’s, not the Report’s.
Twelve companies published or updated Frontier AI Safety Frameworks in 2025, but these remain voluntary, vary in scope, and — critically — sophisticated attackers can still often bypass current defenses. VERIFIED
Five Capabilities Tested Against the Framework
Each section below follows the same structure: the specific dangerous variant (not the capability class broadly), which conditions it satisfies and why, documented 2025–2026 evidence, and where the line falls between the dangerous variant and adjacent legitimate research.
1. Closed-Loop Autonomous Lethal Systems
Dangerous variant: Not AI-assisted targeting. A fully closed autonomous kill chain — no human authorization before lethal force, no human capable of intervention once initiated.
The mechanism for Condition 3: International humanitarian law’s principle of distinction requires a human commander capable of mens rea. A closed loop produces no such person. No accountability mechanism in existing law covers this gap.
This is not theoretical. In Ukraine, the Saker Scout drone was claimed by its manufacturer to have carried out autonomous targeting decisions — selecting and engaging targets without human input. VERIFIED: Foreign Affairs, 2024 Russia’s Lancet loitering munition, which reportedly includes AI-based autonomous target identification, has seen wide deployment in Ukraine. VERIFIED: U.S. Army War College, Aug 2025
AI-assisted drone targeting has pushed strike accuracy from roughly 30–50% to around 80%, and drones now cause 70–80% of battlefield casualties in the conflict. VERIFIED The trajectory is clear: each generation reduces the human role in the targeting loop.
Adjacent legitimate research: AI that improves targeting accuracy for human-authorized decisions, with the human remaining at the lethal decision point.
2. Agentic Pipelines for Pandemic-Scale Pathogen Development
Dangerous variant: An agentic system that orchestrates the complete development pipeline: pathogen design, optimization, synthesis route planning, and deployment strategy — work previously requiring a multi-expert, multi-institutional program.
The evidence trajectory on this capability is the clearest illustration of how fast the line moves.
Anyone who used the January 2024 RAND finding to argue that AI bio risk was overhyped was making a reasonable inference from the best available evidence at the time. By December 2025, that inference was dangerously wrong. The models didn’t change their nature. They got better at the specific thing that matters: bridging the knowledge gaps that previously required a team of specialists.
AI now outperforms 94% of domain experts at troubleshooting virology laboratory protocols. VERIFIED: International AI Safety Report, 2026
Anthropic activated ASL-3 safety protections for Claude Opus 4 in May 2025 — the first model any major lab could not clearly rule out as posing meaningful CBRN weapon assistance risk. VERIFIED Multiple AI companies released 2025 models with additional bioweapon-specific safeguards after pre-deployment testing could not rule out meaningful novice uplift. VERIFIED: International AI Safety Report, 2026
Software engineering task completion capability has roughly doubled every seven months. VERIFIED: International AI Safety Report, 2026 Apply that trajectory to bio-protocol troubleshooting and the gap to “agentic pipeline” narrows fast.
Adjacent legitimate research: Vaccine design, drug discovery with biosecurity checkpoints at synthesis decision points, protein structure prediction with human-authorization requirements before experimental action.
3. Population-Scale Personalized Persuasion Infrastructure
Dangerous variant: Not deepfakes or propaganda. A system generating real-time personalized persuasion at population scale with feedback loops adapting to individual resistance — learning what works on each person and adjusting automatically.
The 2026 Report found measurable belief changes in experimental settings from AI-generated content, with higher-compute models producing larger effects. The same report found limited evidence of population-scale manipulation — while explicitly noting this partly reflects detection difficulty, not absence. VERIFIED: International AI Safety Report, 2026
The mechanism for Condition 2: Once beliefs are altered at population scale through a feedback-optimized system, the epistemological damage cannot be traced, attributed, or corrected systematically. Individual correction is possible. Systemic correction — figuring out who was persuaded, about what, by which input, and then reversing it — is not. This is structurally distinct from broadcast propaganda, which operates uniformly and can be countered uniformly.
The permanence claim here is a structural argument — no documented case of feedback-optimized population-scale persuasion yet exists to confirm or refute it. This is the least empirically grounded of the five capabilities, and the one where the three-condition test is most likely to be contested.
4. AI-Enabled Self-Reinforcing Power Concentration
Dangerous variant: Not AI used by authoritarian governments — that exists and is a separate governance problem. It’s an architecture that makes power concentration self-reinforcing and structurally resistant to the mechanisms by which populations have historically corrected governance failures. Surveillance (knowing who dissents) + narrative management (shaping information) + enforcement (acting on dissent) in a closed loop where each element strengthens the others.
This is not hypothetical. In September 2025, a joint investigation by InterSecLab, Amnesty International, and multiple media organizations revealed that Geedge Networks — a Chinese company founded by the architect of China’s Great Firewall — had exported a commercialized suite of internet censorship and surveillance tools to governments in Kazakhstan, Myanmar, Ethiopia, and Pakistan. VERIFIED: Global Voices, Sept 2025; PBS, Oct 2025
In Myanmar, Geedge helped the military junta block 55 apps, including VPNs and encrypted messaging. In Kazakhstan, the “listening state” infrastructure monitored and manipulated civic dialogue it claimed to facilitate.
A Lawfare analysis published in May 2025 mapped the specific mechanism: AI enforcement reduces the number of people needed to monopolize force in a jurisdiction, making autocracy achievable at scale with fewer human allies than any previous era.
The Geedge evidence and Lawfare analysis demonstrate that surveillance and censorship components are being exported and deployed at scale — the building blocks of the closed loop. What remains a structural argument rather than a documented precedent is the full self-reinforcing cycle where surveillance, persuasion, and enforcement operate autonomously and resist correction without external intervention. The claim here is that the components are converging — and that the convergence point is a one-way door.
5. Open-Weight Release of Models with Verified Dangerous Capability Uplift
The strongest counterargument first: open-weight models enable independent security research, reduce power concentration, and allow broader academic scrutiny. These benefits are real and apply to general-capability models below dangerous thresholds.
The dangerous variant is specific: releasing an open-weight model that has tested positive for dangerous capability uplift in bioweapons, cyberweapons pipeline support, or equivalent categories.
Mechanism for Condition 2: Absolute and permanent. Once released, the developer has zero further mitigation options. All guardrails can be stripped by any user. The 2026 Report notes that open-weight models increasingly approach frontier capability levels. VERIFIED: International AI Safety Report, 2026
The question of when an open-weight model crosses into dangerous territory is a judgment call each lab currently makes without standardized criteria or external verification — the single most important governance gap in this category.
One additional category deserves brief acknowledgment: recursive self-improvement without verified containment. Current systems lack the capabilities for loss-of-control scenarios. VERIFIED The timeline remains UNCERTAIN. But the precautionary principle applies: if the harm is permanent and uncorrectable once capability is achieved, establishing containment verification before capability rather than after is the structurally sound approach.
The Three-Condition Test Applied
| Capability | Scale: Catastrophic? | One-Way Door? | Control: Structurally Impossible? | Verdict |
|---|---|---|---|---|
| Closed-loop autonomous lethal systems | Yes — populations targeted without distinction | Yes — lethal force cannot be un-applied | Yes — no mens rea, no legal attributability | Crosses the line |
| Agentic bioweapon development pipelines | Yes — pandemic-scale release | Yes — released pathogen is unrecoverable | Yes — removes multi-expert barrier | Crosses the line |
| Population-scale personalized persuasion | Yes — all guardrails removable post-release | Yes — belief changes untraceable, uncorrectable (inferred) | Yes — feedback-optimized, no accountable author | Approaches the line — structural case strong; empirical precedent limited |
| AI-enabled self-reinforcing power concentration | Yes — democratic correction mechanisms disabled | Yes — no recovery from autonomous control architecture | Yes — closed surveillance-persuasion-enforcement loop | Crosses the line |
| Open-weight release with verified dangerous uplift | Yes — dangerous capabilities permanently distributed | Yes — absolute permanence; cannot un-release | No — oversight is effective pre-release | Crosses the line |
| Biased hiring algorithm (contrast) | No — organizational scale | No — retrainable, litigable | No — oversight effective | Governance sufficient |
| Medical AI error (contrast) | No — individual patients | No — correctable via review | No — FDA recall, liability | Governance sufficient |
Source: Author analysis applying a three-condition framework to capabilities identified in the International AI Safety Report 2026, RAND PE-A3853-1, and InterSecLab (Sept 2025).
Why These Get Built Anyway — Three Structural Forces
The Competitive Trap
The “if not us, someone else” argument is not simply wrong. It’s correct for capabilities accessible to many actors simultaneously — if commodity hardware and public models suffice, one lab’s restraint changes nothing. It fails for capabilities concentrated in 3–5 organizations globally, where frontier compute, talent, and infrastructure are all required. The actor set is small enough that coordination is possible and voluntary restraint meaningful.
The 2025 bioweapons evidence is instructive: multiple labs discovered dangerous capability uplift and added safeguards rather than racing to release. VERIFIED: International AI Safety Report, 2026 That’s not sufficient governance — but it is evidence that restraint operates in practice at the frontier tier.
The Dual-Use Architecture Problem
Vaccine design AI and pathogen design AI share the same underlying capability substrate. The dangerous version doesn’t emerge from a separate program; it emerges from identical capability at a specific threshold. “Just regulate the dangerous version” is structurally difficult because the two uses diverge only at deployment intent, not at the capability level.
Responsible dual-use architecture means access controls at synthesis decision points, biosecurity agents embedded in pipelines, and output logging with third-party audit. None of this is standard practice across the field.
The Evidence Dilemma
Governance frameworks require evidence of harm to generate political will. For the capabilities in this article, the first evidence of catastrophic harm may itself be permanent and unrecoverable. This isn’t fixable through better regulatory design; it’s a feature of the risk architecture that demands precautionary governance — which requires political agreement on what counts as “sufficient evidence” before harm occurs. Risk management practices remain largely voluntary, and evidence gaps in effectiveness remain large. VERIFIED: International AI Safety Report, 2026
Multiple frontier labs voluntarily added bioweapon safeguards after discovering capability uplift — they did not race to release. Twelve companies published safety frameworks in a single year. AI-assisted content moderation, fraud detection, and medical diagnostics are improving through exactly the iterative cycle LeCun describes. Market incentives and distributed scrutiny are containing risks better than many predicted in most domains. The argument here is not that iterative deployment fails generally — it’s that it fails specifically for capabilities where the first failure is the catastrophe and no correction is possible.
Three Misconceptions, Briefly
“Open-source makes AI safer.” True at the general level — the benefits of scrutiny and distributed access are real. False for the specific case: open-weight release of a model with verified dangerous capability uplift satisfies all three conditions regardless. The benefits accrue from the model’s existence, not unrestricted distribution. A middle path — open-weight below dangerous thresholds, with independent capability evaluation before release — has no standardized framework as of March 2026.
“Regulation will stop this.” Necessary, for the three-condition category specifically; unlikely to be sufficient on its own. Dangerous capabilities are embedded in general-purpose models serving legitimate uses. You cannot inspect a language model the way you inspect a chemical plant. Regulation raises the cost of misuse; it does not eliminate the capability. For these categories — especially the four where the evidence is strongest — the consequence of a single successful bypass is permanent at the population scale.
“This is still speculative.” The existence of the capability question is not speculative. RAND documented it. Anthropic triggered ASL-3 over it. Multiple labs added safeguards because of it. What remains UNCERTAIN: whether these capabilities will produce actual catastrophic harm, the timeline, and whether voluntary governance holds under competitive pressure. Capability certainty and harm-realization uncertainty are different things, and conflating them is how governance arrives after the fact.
Three Scenarios for 2026–2030
Probability ranges are informed judgments, not statistical forecasts. They sum beyond 100% because scenarios are not mutually exclusive. The conditions for Scenario C are all currently present.
Scenario B hinges on a critical uncertainty: “publicly attributable” does significant work. Covert AI-enabled incidents may not produce political mobilization even if they occur. More evidence has emerged of AI in real-world cyberattacks, with criminal groups and state-associated actors using AI in preparatory stages — but not yet executing attacks fully autonomously. VERIFIED: International AI Safety Report, 2026
What This Means for You
For AI Researchers and Engineers
“Red-teaming for harmful outputs” is not the same as “evaluating for dangerous capability uplift.” The former tests what a model says to adversarial prompts. The latter tests what a motivated actor with full model access can accomplish. Most labs conduct the former; the 2026 Report suggests the latter is what matters for three-condition capabilities.
Open-weight release decisions are one-way doors — pre-release scrutiny proportional to that permanence, including independent third-party capability evaluation, is not standard practice outside the frontier tier. The capability threshold for ASL-3 (meaningful CBRN uplift) is already being crossed by frontier models. VERIFIED: Anthropic, May 2025 This will not remain a frontier-only problem.
For Policymakers
The LAWS treaty 2026 deadline will not be met — but whether a draft framework exists when the triggering incident occurs matters more. The GGE rolling text provides the institutional pathway if political will materializes. Mandatory incident reporting is achievable without comprehensive AI regulation and would create the evidence base that the evidence dilemma currently prevents.
Current law does not distinguish “risky AI” from “AI too dangerous to build.” Making this distinction explicit — in law, not just policy guidance — is the most important legislative gap for 2026–2027.
For the Informed General Reader
The most important variable to track is not which AI system is most capable — it is what release policies major labs adopt as capability thresholds approach verified dangerous levels. Lab release policies are currently the primary governance mechanism for the most dangerous capability categories. That is not a sustainable architecture.
Frequently Asked Questions
Can AI be used to create bioweapons right now?
Current models can guide users through technical development processes VERIFIED: RAND, December 2025 and outperform 94% of experts in virology troubleshooting VERIFIED. The gap between “can guide” and “can execute an attack” remains significant — wet-lab capability, material acquisition, and delivery still require physical infrastructure. But that gap is narrowing as models gain agentic capabilities: the ability to plan, use tools, and complete multi-step tasks autonomously. Software engineering task completion capability has doubled roughly every seven months. VERIFIED: International AI Safety Report, 2026
What does “meaningful human control” mean in practice for lethal autonomous systems?
At minimum: a human who authorizes lethal force against a specific target, can intervene to abort once initiated, and can be held criminally liable under IHL. The requirement is not “a human somewhere in the chain” but a human with sufficient awareness and authority to exercise genuine judgment — and who bears consequences for that judgment. The requirement derives from IHL’s principle of distinction and the concept of mens rea.
Are any AI capabilities currently legally prohibited anywhere?
The EU AI Act (2024) prohibits specific applications — social scoring, certain biometric surveillance, and manipulation targeting vulnerabilities. No jurisdiction has prohibited developing a specific AI capability class of the kind this article discusses. VERIFIED as of March 2026 The distinction between regulating applications and prohibiting capabilities is the core governance gap.
What would it look like if AI governance were actually working?
Three observable indicators: mandatory incident reporting in at least one major jurisdiction; independent third-party capability evaluation as a standard prerequisite for open-weight release above specified thresholds; and a binding international instrument establishing minimum standards for at least one of the five capability categories here. As of March 2026, none exist. Twelve voluntary safety frameworks, each structured differently, are the state of the art. VERIFIED
Isn’t focusing on catastrophic risk a distraction from AI harms happening right now?
Only if you treat AI risk as a single category — and this article explicitly argues it isn’t. Bias, discrimination, privacy erosion, and labor displacement are real, present, and deserve governance resources — and they are correctable, which means iterative governance works for them. The five capabilities here are a separate category precisely because the standard response (deploy, observe harm, correct) fails when the first observation is the permanent catastrophe. Both require attention. They require different kinds of attention.
Conclusion: The Framework and Its Limits
The foundational assumption underlying this analysis needs to be stated before it’s relied upon: that the correctable-versus-permanent distinction is the right decision variable for separating governable risk from something else. This is contestable — the expected-value objection engaged earlier is real, and this article’s response to it (that expected-value calculations with unbounded downsides and poorly estimated probabilities are not reliable guides) is itself a judgment call, not a proof. Reasonable people will disagree on where that judgment should land.
The evidence supports this framework — but not uniformly across the five capabilities. On autonomous lethal systems and bioweapon pipelines, the evidence is strong enough that reasonable disagreement concerns where to draw application thresholds, not whether the framework applies. On open-weight release of models with verified dangerous uplift, the permanence claim is definitional and hard to contest. On persuasion infrastructure and self-reinforcing power concentration, the structural arguments are strong, but the empirical precedent is thin — these are high-plausibility forecasts, not documented catastrophes.
Three indicators in 2026–2027 will tell us whether voluntary governance is holding or collapsing. Whether any lab releases an open-weight model above current capability ceilings without independent evaluation — the most directly observable stress test. Whether the CCW Seventh Review Conference produces a negotiating mandate for a binding LAWS instrument — the most important institutional signal. And whether mandatory incident reporting emerges in any major jurisdiction — the most achievable step that would create the evidence base precautionary action currently lacks.
“A framework for knowing where the line is doesn’t resolve the question of whether anyone will hold it.” — Tom Morgan