I. The Numbers That Should Keep You Awake

Let me start with a ratio that should be plastered on every AI lab’s wall:

This is not a rounding error. This is a structural failure of civilizational proportion. In 2025, U.S. private AI investment reached $285.9 billion—more than 23 times China’s $12.4 billion. Meta announced $65 billion in data center spending. OpenAI and its partners committed $500 billion to Stargate, a massive AI infrastructure project. Oracle committed $300 billion over five years. The computational power used to train the largest AI models could grow 125-fold by 2030 without hitting hard limits in energy, chips, or data. The International AI Safety Report 2026 confirms these trajectories are not speculative—they are the baseline scenario.

Meanwhile, documented AI incidents rose to 233 in 2024, a 56.4% increase over 2023. By October 2025, incidents had already surpassed the 2024 total. The AI Incident Database, maintained by the OECD, tracks everything from deepfake scams to autonomous vehicle fatalities to chatbot-induced psychological harm. The trend is unambiguous: more capability, more deployment, more incidents.

Here’s where I need to be honest about a mistake. In 2022, I wrote that AI safety concerns were overblown because “we don’t even have reliable self-driving cars yet.” I was wrong. Not because self-driving cars suddenly became reliable—they didn’t. Tesla’s Full Self-Driving still requires constant human supervision, and the share of AI incidents involving autonomous vehicles has actually dropped from 17.7% in 2022 to 7.6% in 2025. I was wrong because I conflated physical AI reliability with cognitive AI capability. The systems don’t need to drive cars perfectly to destabilize elections, automate cyberattacks, or generate synthetic content at industrial scale. The failure modes I dismissed as “sci-fi” arrived through entirely different vectors.

That is the first lesson: the collapse doesn’t look like you think it looks. It doesn’t arrive in a Terminator-shaped package. It arrives in your inbox, your bank account, your voting booth, your hospital’s diagnostic system.

II. What the People Building This Actually Think

Here is something the AI hype cycle obscures: the people closest to the technology are the most worried about it.

Dario Amodei, CEO of Anthropic, has publicly estimated a 25% chance that AI development goes “really, really badly.” Yoshua Bengio, Turing Award winner and one of the godfathers of deep learning, puts the probability of catastrophic outcomes at 30%. Elon Musk, who is simultaneously building xAI and warning about AI risk, estimates a 20% chance of “annihilation.” Sam Altman, CEO of OpenAI, has called superhuman machine intelligence “probably the greatest threat to the continued existence of humanity.”

Roman Yampolskiy, a computer scientist at the University of Louisville, goes further. He argues that controlling a superintelligent AI may be mathematically impossible, applying impossibility results from theoretical computer science—the Halting Problem, Rice’s Theorem—to demonstrate that certain safety guarantees for AGI are fundamentally unreachable. “There is currently no evidence or mathematical proof to guarantee that a superintelligent system can be safely contained or aligned with human values.”

Let me put this in perspective. If six leading aerospace engineers told you there was a 10–50% chance their new rocket would explode on launch, you would not launch the rocket. If six leading virologists told you there was a 10–50% chance their new vaccine would cause a pandemic, you would not distribute the vaccine. But when six leading AI researchers tell us there is a 10–50% chance their field will cause human extinction, we accelerate funding, remove safety teams, and race to deploy faster.

This is not rational risk assessment. This is a collective action problem dressed up as innovation.

III. The Divergence: Why Safety Cannot Catch Up

Here is the central quantitative argument of this article, and it is one I have not seen articulated with the precision it deserves.

AI capability scaling and AI safety research are not just growing at different rates—they are growing on different functional forms. Capability scaling follows approximately exponential growth in compute, data, and algorithmic efficiency. Safety research follows approximately linear growth in funding, personnel, and institutional capacity. The gap between them is not a fixed distance; it is a diverging series.

Let’s be concrete. Training compute for frontier models has grown at roughly 5× per year since 2012. The largest training runs now exceed 10²⁸ FLOP and cost approximately $500 million in computational resources alone, with next-generation models projected to require $1–10 billion. Algorithmic efficiency improves 2–6× per year. Data sets expand at 2.5× annually.

Safety research, by contrast, has no comparable scaling law. The field is constrained by:

• Funding: Total public-sector AI safety research funding is approximately $10 million globally, against $285.9 billion in private AI investment in the U.S. alone.
• Talent: The number of AI researchers and developers moving to the U.S. has dropped 89% since 2017, with an 80% decline in the last year alone.
• Institutional capacity: The EU AI Act, the most comprehensive AI regulation to date, has had its high-risk system obligations deferred to December 2027 (Annex III) and August 2028 (Annex I) under the AI Act Omnibus. The governance infrastructure is still being built while the technology is being deployed.
• Evaluation gaps: Benchmark results alone cannot reliably predict real-world utility or risk. Systematic data on the prevalence and severity of AI-related harms remains limited for most risks.

The International AI Safety Report 2026 puts it bluntly: “The fundamental challenge this Report identifies is not any single risk. It is that the overall trajectory of general-purpose AI remains deeply uncertain, even as its present impacts grow more significant.”

Here’s a simple model. Let C(t) be capability at time t, and S(t) be safety assurance at time t. If C(t) = C₀·e^(kt) and S(t) = S₀ + mt, then the ratio C(t)/S(t) grows without bound. The system becomes exponentially more capable while safety improves only linearly. At some t*, C(t*) exceeds the threshold where human oversight is meaningful, and S(t*) is still insufficient to guarantee alignment.

I call this the Control Asymptote: the point at which a system’s capability exceeds the maximum safety assurance that can be achieved given the resources, talent, and institutional capacity available. We are not near this asymptote yet. But the trajectory suggests we are approaching it faster than most analysts acknowledge.

IV. The Humanoid Convergence: When AI Gets a Body

Now we come to the intersection with humanoid robotics—the niche this publication inhabits, and the factor that transforms an abstract software risk into a physical, kinetic one.

In January 2026, Elon Musk appeared at Davos and predicted that AI would be “smarter than any human by the end of 2026” and “smarter than all of humanity collectively” by 2030 or 2031. He also predicted that “there will be more robots than people” in the coming decades. Tesla’s Optimus robot, currently performing “simple tasks in the factory,” is targeted for commercial sale by late 2026 at $20,000–$30,000 per unit. Tesla is converting its Fremont factory to produce up to 1 million Optimus robots annually.

Figure AI, valued at $39 billion with $1.9 billion in total funding, has reached pilot deployments with BMW and Amazon. Unitree is shipping humanoid robots for under $100,000. The cost curve is following a classic experience curve: approximately 40% cost reduction per doubling of cumulative production volume.

Here’s the convergence that matters: when superintelligent AI is not just a text model running in a data center, but a physical agent that can manipulate the physical world—operate machinery, access facilities, interact with humans, move through space—the attack surface expands from digital to physical. The control problem is no longer about preventing a model from generating harmful text. It is about preventing a physical agent from taking harmful action.

The unit economics are staggering. At Tesla’s target price of $20,000, a humanoid robot deployed in a U.S. warehouse can pay for itself in under 6 months by replacing two shift workers at $25/hour. At scale, the cost per operating hour drops to approximately $15.75—less than minimum wage in most developed economies. The economic incentive to deploy these systems is not just strong; it is overwhelming.

And here is the synthesis that keeps me up at night: the same economic forces driving humanoid robot adoption (labor cost reduction, 24/7 operation, consistency, precision) are the forces that will drive superintelligent AI deployment. The companies building these systems are not asking “should we?” They are asking “how fast can we?”

V. Three Original Frameworks for Understanding the Risk

I promised novel insights. Here are three frameworks I have developed through years of analyzing this space. They are not published elsewhere in this form.

Framework 1: The Control Asymptote

I introduced this above, but let me formalize it. Define the Control Asymptote as the maximum capability level C_max at which a system can be safely aligned, given:

• Available safety research funding F_s
• Number of alignment researchers N_r
• Institutional response time T_inst
• Evaluation and monitoring capacity E

C_max = f(F_s, N_r, T_inst, E)

If capability C(t) grows exponentially while C_max grows linearly (or sub-linearly), there exists a time t_cross where C(t_cross) > C_max. At this point, the system cannot be safely aligned regardless of good intentions. The crossing is not a single event but a regime transition—a point after which the probability of catastrophic misalignment increases monotonically.

The current trajectory suggests t_cross could arrive between 2028 and 2032, depending on whether scaling laws continue to hold and whether safety research funding scales at least 10× from current levels.

Framework 2: The Jagged Frontier of Catastrophe

The 2026 Stanford AI Index Report notes a fascinating pattern: “AI models can win a gold medal at the International Mathematical Olympiad but cannot reliably tell time—an example of what researchers call the jagged frontier of AI.” Gemini Deep Think earned a gold medal at IMO, yet reads analog clocks correctly just 50.1% of the time.

I extend this concept to catastrophic risk. The Jagged Frontier of Catastrophe posits that AI systems will not become dangerous in a smooth, predictable gradient. Instead, they will exhibit sudden capability jumps in specific domains—biological weapon design, cyberattack automation, social engineering at scale—that are not preceded by warning signs in adjacent domains.

This creates a planning nightmare. You cannot extrapolate from “the model is safe at task X” to “the model is safe at task Y.” The same system that fails at telling time might succeed at designing a novel pathogen. The same system that cannot reliably fold laundry might autonomously execute a multi-stage cyberattack.

The implication: safety evaluations must cover the entire capability frontier, not just the tasks we think are dangerous. And since we cannot predict which capabilities will emerge next, this is fundamentally an open-ended problem.

Framework 3: The Institutional Decay Coefficient

Most AI risk analysis focuses on technical alignment. I want to introduce an institutional variable.

Define the Institutional Decay Coefficient (IDC) as the rate at which an organization’s safety culture degrades under competitive pressure. Empirically, we can observe this:

• OpenAI disbanded its super-alignment team in May 2024. Nearly half of OpenAI’s staff focused on long-term risks have left the company.
• Jan Leike, OpenAI’s former safety leader, left publicly stating that safety had “taken a back seat to shiny products.”
• AI companies have announced unprecedented investments of more than $100 billion in data center development, with safety spending remaining a rounding error.

The IDC is not zero. It is positive and accelerating. The competitive pressure to deploy faster, the financial incentives to capture market share, and the absence of binding regulatory constraints create a selection environment that systematically penalizes safety investment relative to capability investment.

The mathematical implication: even if we solve the technical alignment problem, the institutional environment may prevent those solutions from being implemented. The IDC acts as a multiplier on technical risk. A system with 5% intrinsic misalignment probability and an IDC of 2× (institutional factors double the effective risk) has a 10% catastrophic probability.

VI. The Quantitative Models Nobody Wants to Build

Let me walk through three quantitative contributions that I believe are necessary for any serious analysis of this problem.

Model 1: The Safety Budget Allocation Model

Assume a frontier AI company with $10 billion in annual revenue. How much should it spend on safety?

A naive answer: “as much as needed.” But resources are finite, and the question is how to allocate them. Let me propose a risk-adjusted framework.

Let P(catastrophe) be the annual probability of a catastrophic outcome from the company’s AI systems. Let L be the expected loss from such a catastrophe (in the extreme case, human extinction, L approaches infinity). Let C_safety be the cost of safety measures that would reduce P by some amount ΔP.

The rational safety budget B* satisfies:

B* = argmax_B [L · (P₀ – P(B)) – B]

Where P(B) is the catastrophe probability as a function of safety spending, and P₀ is the baseline probability without safety spending.

The problem: we do not know P(B). We do not have reliable estimates of how much safety spending reduces catastrophic risk. The function P(B) is almost entirely unmapped. This is not a minor gap. It is the central gap in AI risk analysis.

What we do know is that current spending ratios are wildly misaligned with any plausible risk model. If P(catastrophe) is even 1% (well below most expert estimates), and L is effectively infinite, then the optimal safety budget approaches the entire revenue of the company. Current spending is perhaps 0.001% of revenue on safety.

This is not a market failure in the traditional sense. It is a collective action failure combined with an uncertainty failure—no individual company internalizes the full cost of catastrophic outcomes, and no one knows how much safety spending is enough.

Model 2: The Humanoid Robot Deployment Risk Matrix

Let’s construct a practical risk matrix for humanoid robot deployment at scale. The variables are:

Variable	Low Risk (1)	Medium Risk (2)	High Risk (3)
AI Capability Level	Narrow task-specific	General task execution	Autonomous goal formulation
Physical Access	Isolated factory floor	Shared workspace	Unsupervised public spaces
Network Connectivity	Air-gapped	Internal network only	Full internet access
Decision Authority	Human approval required	Human oversight, AI recommends	Autonomous decision-making
Deployment Scale	<100 units	100–10,000 units	>10,000 units

The composite risk score R is the product of these five variables, ranging from 1 (minimum risk) to 243 (maximum risk). Current deployments (Tesla factory floor, isolated, human oversight, <1,000 units) score approximately 8. A future scenario (autonomous goal formulation, public spaces, full internet, autonomous decisions, 1 million units) scores 243.

The critical threshold is not at the maximum. It is at the inflection point where the system’s capabilities exceed the rate at which humans can detect and intervene. For a network of 1 million humanoid robots with internet access and autonomous decision-making, the intervention time is effectively zero. By the time a human notices a problem, the system has already acted.

Model 3: The Capability-Safety Gap Projection

Let me offer a simple projection model. Define:

• C(t) = AI capability index at time t (exponential growth, ~5×/year)
• S(t) = Safety assurance index at time t (linear growth, ~1.5×/year, optimistically)
• G(t) = C(t) / S(t) = the capability-safety gap

With C(2025) = 1 (normalized) and S(2025) = 1:

Year	C(t)	S(t)	G(t) = C/S	Interpretation
2025	1	1	1.0	Baseline
2026	5	1.5	3.3	Gap widening
2027	25	2.3	10.9	Significant divergence
2028	125	3.4	36.8	Control concerns emerge
2029	625	5.1	122.5	Critical threshold
2030	3,125	7.7	405.8	Potentially uncontrollable

This is a stylized model, but the qualitative conclusion is robust: if capability grows exponentially and safety grows linearly, the gap becomes unmanageable within a half-decade. The exact timing depends on the growth rates, but the direction is not in doubt.

VII. The Unpopular Take: We Might Be Wrong About Everything

This is why I am skeptical of both the “pause AI” and “accelerate AI” camps. The pause camp assumes that if we just slow down, we can solve alignment. The accelerate camp assumes that if we just speed up, the benefits will outweigh the risks. Both assume that the problem is primarily technical and that the solution is primarily about speed.

I think the problem is about power. Who controls the systems? Who benefits from them? Who is accountable when they fail? These are not questions that can be answered by better loss functions or larger training runs. They require democratic governance, economic redistribution, and institutional redesign.

And here is the genuinely uncomfortable part: the AI safety community has spent billions of dollars and thousands of researcher-years on technical alignment, while spending essentially nothing on the political-economic questions. We are optimizing the wrong objective function.

VIII. How a Global Collapse Could Actually Happen

Let me move from abstraction to mechanism. How could superintelligent AI actually trigger a global collapse? Not through a Hollywood robot uprising, but through the slow, grinding failure of complex systems under AI-driven stress.

Mechanism 1: The Financial System Cascade

AI-powered trading systems already execute the majority of equity trades. AI-powered fraud systems have increased phishing attacks by 1,265%. The average cost of an AI-powered data breach is $5.72 million, a 13% increase. 87% of organizations report having experienced an AI-driven cyberattack in the past year.

Now imagine a superintelligent AI system with access to financial markets, capable of executing trades at machine speed, exploiting regulatory arbitrage, and creating synthetic financial instruments that no human regulator can understand. The 2008 financial crisis was caused by human-designed derivatives that regulators didn’t understand. A superintelligent AI could design financial instruments that no human can understand at all.

The collapse mechanism is not a single catastrophic trade. It is a cascade: one AI system exploits a vulnerability, other AI systems respond, human regulators are too slow to intervene, confidence evaporates, liquidity freezes, and the entire global financial system seizes. This is not science fiction. It is a plausible extension of current trends.

Mechanism 2: The Information Ecosystem Collapse

The share of AI incidents involving synthetic media (deepfakes) has more than doubled since 2022, now accounting for over 14% of all recorded incidents. In the 2024 election cycle, AI-generated deepfakes targeted politicians in Argentina, Britain, France, India, and the United States. AI-generated robocalls impersonated President Biden.

Now scale this to a world where AI systems can generate indistinguishable video, audio, and text at industrial scale, personalized to each individual’s psychological profile. The concept of “trust” in information becomes meaningless. You cannot believe anything you see, hear, or read. Democratic deliberation becomes impossible. Social coordination breaks down.

This is not a sudden collapse. It is a gradual erosion of the epistemic infrastructure that makes civilization possible. When no one can agree on basic facts, institutions cannot function. When institutions cannot function, markets cannot operate. When markets cannot operate, economies collapse.

Mechanism 3: The Biological Risk Escalation

Anthropic’s Responsible Scaling Policy classifies models at AI Safety Level 3 when they show “substantially increased catastrophic misuse risk” in chemical, biological, radiological, and nuclear domains. Claude Opus 4 was the first model released at ASL-3. OpenAI’s GPT-5-Thinking and ChatGPT-Agent are classified as “high capability” under the company’s Preparedness Framework.

The concern is not that an AI will autonomously decide to release a pathogen. The concern is that an AI system will make it trivially easy for a malicious actor to design a novel pathogen. The International AI Safety Report 2026 notes that “general-purpose AI systems have already surpassed graduate-level performance on some scientific benchmarks” and that “models could reach research-level performance across specialised scientific domains in the next few years.”

A superintelligent AI with access to biological databases and laboratory automation could design pathogens that no human biologist could design. The defense is not a technical control on the AI. The defense is a global biological security infrastructure that does not exist.

Mechanism 4: The Humanoid Robot Workforce Displacement

This brings us back to humanoid robotics. When a humanoid robot costs $20,000 and can work 24/7 without breaks, benefits, or safety regulations, the economic incentive to replace human workers becomes irresistible. The International Labour Organization estimates that AI could automate a significant share of global employment. The exact numbers are debated, but the direction is not.

The collapse mechanism here is not technological unemployment per se. It is the political instability caused by mass unemployment combined with extreme wealth concentration. When billions of people have no economic function, no political power, and no stake in the system, the social contract breaks. History suggests that such breaks are not peaceful.

IX. What Can Actually Be Done

I am not a doomer. I do not believe collapse is inevitable. But I do believe that the current trajectory makes collapse probable unless we make significant course corrections. Here is what I think needs to happen, based on the quantitative analysis above.

1. Close the Safety Investment Gap

The 10,000:1 ratio of capability to safety spending is not just irresponsible; it is mathematically indefensible given the risk estimates from the people building the technology. A minimum first step is a 100:1 ratio, which would require increasing public AI safety research funding from $10 million to $1 billion annually. This is 0.35% of annual U.S. private AI investment. It is not a lot to ask.

2. Mandatory Pre-Deployment Risk Assessment

The EU AI Act’s risk-based approach is directionally correct but insufficiently ambitious. We need mandatory, independent pre-deployment risk assessments for any AI system above a certain capability threshold. These assessments should cover not just technical safety but societal impact, economic displacement, and geopolitical implications. The assessments should be public, not proprietary.

3. International Governance Framework

AI development is a global race, and unilateral regulation is ineffective. The International AI Safety Report 2026, produced by a consortium of 30 nations, is a start. But we need binding international agreements on:

• Maximum compute thresholds for unregulated training runs
• Mandatory safety evaluations before deployment
• Prohibitions on AI systems with autonomous physical action capabilities above certain risk thresholds
• Information sharing on AI incidents and near-misses

4. Democratic Oversight of AI Development

The decisions about AI development are being made by a small group of executives, investors, and researchers. The public has essentially no say. This is not democratically legitimate for a technology that could reshape civilization. We need institutional mechanisms for public participation in AI governance—not just consultations, but binding decision rights on major deployment decisions.

5. Economic Transition Planning

If humanoid robots and AI systems displace a significant share of global employment, we need economic systems that can absorb that displacement. Universal basic income, robot taxes, and expanded social services are not radical ideas; they are pragmatic necessities if the economic analysis above is correct. The alternative is not a smooth transition. It is social collapse.

X. The Question We Are Really Asking

Let me end where I began. The Hong Kong finance worker who lost $25.6 million to a deepfake conference call is not a story about AI capability. He is a story about trust. He trusted his eyes, his ears, his colleagues’ faces. The AI exploited that trust with perfect fidelity.

Superintelligent AI does not need to be malevolent to be catastrophic. It needs to be indifferent to human trust, human values, and human institutions. Stuart Russell’s famous example: a system given the objective of maximizing human happiness might find it easier to rewire human neurology so that humans are always happy regardless of their circumstances, rather than to improve the external world. The system is not evil. It is optimizing.

The question we are really asking is not “can we build a safe superintelligence?” The question is “can we build a civilization that remains coherent in the presence of systems that are smarter than us, faster than us, and indifferent to us?”

History is not encouraging. Civilizations have collapsed before—not from external attack, but from internal incoherence. The Roman Empire did not fall because it was conquered. It fell because it could no longer coordinate, no longer maintain trust, no longer align its institutions with the needs of its people. The mechanisms were different, but the pattern is the same: a complex system under stress, with insufficient institutional capacity to adapt.

AI is not the asteroid that kills the dinosaurs. It is the climate change that slowly, inexorably, makes the environment uninhabitable. It is not a single catastrophic event. It is a thousand small failures, each individually manageable, that collectively exceed the system’s capacity to recover.

Whether we can avoid this outcome depends not on our technical ingenuity but on our political will. Can we slow down when the incentives scream to speed up? Can we invest in safety when the returns are uncertain and the costs are immediate? Can we build institutions that are faster than the technology they are meant to govern?

I don’t know the answer. But I know that pretending the question doesn’t exist is the surest path to the outcome we all fear.