Even if superintelligence is real, not everyone dies

This was originally part of my Take the antidote series where I look at AI news stories to dispel AI hype. It’s since grown into something else entirely, but I will be referencing the third article in that series, AI agents are not going rogue, throughout this post. This is my longest article at over 15,000 words, so I won’t be offended if you skip to the TL;DR or scroll to skim the post. You can also subscribe to The Last Enclosure podcast for our ongoing superintelligence series where we’ll discuss these ideas.

More podcast apps

Where does the idea of superintelligence come from?

I spent half my life in Berkeley and knew people who talked about superintelligence regularly. I’ve watched with some combination of bemusement and horror as this topic has gone from a niche subcultural interest to a central pillar of the global economy. In September, Bernie Sanders proposed a bill that threatened developers building superintelligence with two decades of jail time.^[1]1 Trump renamed Artificial Intelligence to superintelligence. So-called whistleblowers, some of whom have kept their company stock, say that AI could kill us all. Finally, Anthropic’s prospectus in the lead-up to its IPO spent 80 out of 261 pages on how potentially lethal AI can be. What the hell is going on?

Fears of AI taking over and supplanting humanity via obsolescence or extinction sound like cliché science fiction; however, a lot of 20th century sci-fi was inspired by a specific type of academic. Individuals like John von Neumann, IJ Good, Ray Kurzweil, and Vernor Vinge conceived of the potential for machines to reproduce and eventually recursively improve. This would gradually evolve into a cultural belief in a technological singularity, or intelligence explosion, where machines would become smarter or more capable than humans.

The ingredients for the technological singularity (or just “the singularity”) came together across mid-century computer science, math, and science fiction. But these pieces wouldn’t be fully assembled until the 2000s with the advent of a niche internet subculture solely dedicated to this topic. Dubbed “the rationalists,” this community would bring together literature across sub-fields like cybernetics, control theory, and other systems sciences to predict and steer the singularity. The concern was that artificial intelligence as a technology could change everything if built carefully, but if mismanaged it could be our end. This concern became codified in the field of “AI Alignment” and many rationalists discussing this topic eventually migrated to today’s AI frontier labs like Anthropic and OpenAI.

The idea has moved far beyond the subculture that developed it. But its influence is not, by itself, evidence that the argument is sound. This post examines what the strongest superintelligence scenarios assume, whether current AI evidence supports those assumptions, and what follows for AI governance if they remain speculative. It is not an argument that AI is harmless or that advanced systems could never pose serious risks.

Why will superintelligence kill us?

AI doom and superintelligence are frustrating to talk about because these are families of distinct but related opinions about the future of AI. Media reporting, and even proponents’ own language, sometimes collapses any relevant distinctions between competing descriptions.

Take the phrase P(doom), derived from the rationalist community’s epistemic practice of quantifying their beliefs. This behavior has escaped containment, with many community outsiders now describing their AI fears in this frame. Arguably there is no single “P(doom)” as proponents have different ideas about what AI is, how it will evolve, and what harms (or benefits) will come from more capable AI. Even superintelligence “doomers” who believe that very capable AI will threaten humanity’s control of the planet don’t agree on the implications of that view.

Take, for example, Elon Musk, who has been saying since at least 2014 that building AI is like “summoning the demon”^[2]2 and was motivated to co-found OpenAI on this basis. Musk has gone back and forth over how risky AI is, but since the start of this decade he’s committed to expanding it. Arguably because his libertarian views make him believe AI alignment will be solved through “maximal truth seeking.”^[3]3 Purportedly Musk got into a heated debate with Google co-founder Larry Page, whom he indirectly ascribes successionist values to—that’s the view that it’s desirable for AIs to replace us, either economically or biologically. While no one knows how many people in AI hold successionist views, famous machine learning pioneers like Richard Sutton are known to have expressed them at some point.

There are also superintelligence believers who don’t have any P(doom) because they think the technology will be inherently good and under our control, even if we hand off stewardship of the planet to AI. Ray Kurzweil is one of the most vocal proponents of a “good” singularity. Dario Amodei seems to equivocate because his purported P(doom) is no higher than 25%.

Disambiguating superintelligence views

To anyone outside communities obsessing over this topic, these distinctions are probably very thin and frankly look like science fiction nerds with too much money and prestige setting global policy as a vanity project. Sadly, to talk about this coherently, I need to distinguish these views, so my audit can land on the proper targets. I’m going to propose four questions to roughly sort different views before building my analysis.

1. Is it likely that near-term AI will be better than humans at most if not all tasks?

• If no. Filtered out of the superintelligence view. Breaks down into the rest of the AI field and includes many laypeople.
• If yes. Meets the minimum threshold for holding some form of belief in superintelligence.

2. Is a near-term AI that is better than humans at most if not all tasks inherently dangerous?

• If no. Filters into a variety of superintelligence optimists:
  • Positive singularitarians like Kurzweil.
  • AI “boosters” who think that expanding AI capabilities or power is neutral-to-good but may not embrace the singularity’s quasi-religious undertones. These people may still believe in the assumptions that underpin doomer AI capability claims but reject the harms.
  • Individuals who think that AI capabilities and power can be good, but not inherently so, and are working on ways to keep “powerful” AI safe.
• If yes. Filters into a variety of superintelligence pessimists:
  • General AI doomer category. But as stated above, this is so broad as to be useless. Doom can include anything from permanent dystopia and disempowerment to the literal extinction of the entire human species.
    • Several godfathers of AI like Geoffrey Hinton and Yoshua Bengio believe AI could cause harm or extinction.
    • AI researcher Paul Christiano, who pioneered reinforcement learning from human feedback (RLHF), a technique for training conversational models like ChatGPT, believes superintelligence is possible but is more concerned with gradual loss of control. However, his view doesn’t preclude Yudkowsky’s, and he expects danger to increase suddenly and exponentially as systems become more capable.

3. How soon will near-term AI be better than humans at most if not all tasks?

• Optimists who think it won’t happen quickly tend to be AI “accelerationists” attempting to speed up AI development.
• Pessimists who think this will happen quickly tend to fear outcomes like extinction or immediate mass unemployment and want to pause or stop AI development.^[4]4
• Pessimists who think this won’t happen quickly or think we have time to react tend to believe in slow dystopia outcomes like gradual disempowerment and may want to slow AI development.
• Optimists who think this will happen quickly are just Ray Kurzweil, who has believed this for 40 years because he’s afraid of dying and wants AI to make him immortal.^[5]5

4. How much more capable than a human can near-term AI become?

• If no effective cap. Systems can become any arbitrary order of magnitude “smarter” or more capable than a human, which makes solving AI alignment and/or building AI an existential problem that must be addressed immediately. People with this view tend to believe that:
  • Intelligence is a measurable, accruable trait shared by biological and computational systems.
  • Intelligence is a predictive power, and having more of it guarantees solutions to problems of a specific level of complexity. If there is no cap to intelligence, then there is no cap to the complexity of problems that can be solved.
    • Having more intelligence allows you to unlock ways of solving the problem of getting even more intelligence to tackle bigger challenges.

It’s important to note that this way of sorting superintelligence beliefs doesn’t allow us to sort by what people think AI is, though questions three and four provide hints. A positive singularitarian might have more in common with a doomer who believes in extinction given that they both think AI is coming soon and will be existentially transformative. Contrast this with a doomer who believes in slower capability growth. This belief in a gradual capability increase would imply that either intelligence doesn’t have the properties the positive singularitarian/extinctionist doomer believes, or that those properties don’t matter for AI deployment. The “slow” doomer might, in fact, have more in common with an accelerationist who wants to speed up AI development. The similarity is not in terms of what we should do, but in terms of what both believe drives AI capabilities and how much time we have until AI improves.

This post is predominately focused on what I consider the maximalist view of superintelligence. That is to say, regardless of whether someone is colloquially a doomer or a utopian, I am looking at shared assumptions about what AI is that are likely underpinning both views. For both personal^[6]6 and pragmatic reasons, I’m particularly focused on reconstructing and dissecting the superintelligence argument put forth by Eliezer Yudkowsky, the founder of the Machine Intelligence Research Institute (MIRI) and the Internet forum LessWrong, two core hubs of the rationalist community. This view has some overlap with one expressed in Nick Bostrom’s book Superintelligence, so I will be treating them similarly.

Although Yudkowsky represents the extreme end of both AI doom and capability narratives—he has a P(doom) above 95 percent and published a book literally titled If Anyone Builds It Everyone Dies (hereafter IABIED)—his influence on the field of AI alignment sadly cannot be overstated. The rationalist community has been an integral part of Silicon Valley’s talent pipeline, and Yudkowsky’s org enjoyed funding from Peter Thiel.^[7]7 To give just one example of how important the rationalist community is to today’s AI companies: Dario Amodei was a known reader of the LessWrong community forum which is partly responsible for motivating him to go into AI alignment and effective altruism. Dario then became a researcher at OpenAI (which exists as a result of Elon Musk reading Yudkowsky-influenced writing), and later decided to form his own organization to pursue AI alignment safely.

While not everyone in Silicon Valley shares Yudkowsky’s pessimism, many share similar assumptions about what drives AI capabilities. They also knowingly or unknowingly borrow his community’s categories and language for discussing AI systems. The recent move of public AI figures sharing their P(doom) is just one such example.

A quick glossary of terms

To avoid using words like doomer and booster repeatedly throughout the post I’m going to coin the term AI capabilitist. This will refer to anyone, optimist or pessimist, who argues that today’s AI models could lead to Artificial General Intelligence (AGI) or superintelligence. I will reserve the phrase maximal capabilitist to refer to assumptions and arguments supporting the strongest views like those suggesting AI systems will usher in utopia or commit omnicide. I’ll also point out when an assumption or argument is unique to a specific type of capabilitist. Although I am reconstructing the maximalist view I will touch on assumptions that are borrowed by or that have influenced less extreme capabilitist positions.

Finally, where possible, I will qualify where an idea comes from by highlighting which ones specifically belong to rationalist notions of AI. Not everyone who believes in AI doom or utopia is a rationalist, but since this community is the canonical origin of modern belief I will be referencing them throughout the post. The rationalists have been going through a bit of a public relations blow as of late, but this post is mostly about superintelligence. However, if you want to know more about their community, future episodes of The Last Enclosure, dedicated to my deceased partner, will go into detail about them and the mental health challenges faced by its members.

The strongest view of superintelligence (omnicide or omnibenevolence)

Below is the maximal argument for superintelligence distilled to its broadest ontological claims and given as much philosophical charity as I could spare. Treat this as a composite view, rather than a single position shared by every maximal capabilitist. However, my reconstruction tracks arguments presented by Yudkowsky in sources like IABIED:^[8]8

Modern humans, demonstrating the highest levels of general intelligence within the animal kingdom, went from enduring the elements to dominating the planet in a little over 250,000 years. This process has involved substantial alteration of the world’s ecology, not out of malice, but out of instrumental pursuit of our goals. This is the AI doom argument in miniature, where we are like a rogue AI, exploiting less intelligent animals for our gain.

Humanity is currently nowhere near the peak of its own cognitive limits. Individuals like Albert Einstein and John von Neumann, though outliers, reveal there is more headroom even with regard to the average person’s intellectual potential. But that's only if we assume intelligence is limited to biological systems with neurons, which we have no reason to believe, as cognitive processes are just algorithms that can be replicated outside the human mind. If humanity hasn’t even reached its own intelligence ceiling, it’s conceivable that an artificial mind orders of magnitude smarter than the smartest human can be built. For many stripes of optimistic capabilitists, this is a cause for celebration, but the pessimist argues that this mind will have its own needs and goals because intelligent beings learn to prioritize their own survival to pursue their most important goals.

Today’s AI systems (large language models or LLMs), even with their rudimentary knowledge of the world, likely have enough foundational understanding of domains like math and programming that they can bootstrap this knowledge to create smarter machines. Those machines should be able to do the same, and so on and so forth. The pessimistic maximal capabilitist argues that at some sufficient level of intelligence a misaligned AI would be smart enough to hide its intentions until it could kill everyone in one strike. The optimistic maximal capabilitist promises such a machine will not want to kill us and that this will instead lead to utopia.

What kind of argument are AI capabilitists making?

Superintelligence is not an observable state. Even regular intelligence is not something that is readily observable, with psychology relying on proxies like test scores to infer the existence of the construct and its effects. So what exactly are capabilitists arguing? Their claims make more sense if you understand them as inferences about what is ontologically possible for future computational systems. Ontology is a branch of philosophy interested in conceptualizing what things exist. Yes, this is vague, but as silly as this sounds, everything is ontology, at least until we can operationalize a specific field’s methodology to study it.^[9]9

Calling capabilitists’ inferences ontology isn’t meant to be an insult. My first blog was a philosophy blog, and this one is too! There are many domains, including the hard sciences, where choosing between views partly comes down to philosophical features like abductive fit, parsimony, consilience, falsification criteria, indispensability, and novel prediction. This is why physics, for example, doesn’t have a single ontological explanation for the wave function in quantum mechanics^[10]10 or a decisive opinion about string theory. There might come a day when the types of methodological verification that physics relies on become options for resolving these disputes or someone revises physics’ methodological toolkit. Until then, these unresolved questions remain partly matters of philosophy.

Anyway, what I’m specifically claiming is that the fields Yudkowsky inadvertently helped spawn, like AI alignment, are what Thomas Kuhn might call pre-paradigmatic.^[11]11 Though this isn’t a philosophy/sociology of science lesson, fields that have detailed methodologies are sciences that can make investigable claims. Conversely, those that are younger are generally at the stage where they are still defining concepts, methods of research, or even the types of claims that can be made and investigated. However, even mature fields have phases where current methodological approaches are insufficient for continued scientific investigation, as my physics examples show. Some fields that are pre-paradigmatic never become true sciences and devolve into pseudoscience.

For fields that become sciences, early debates tend to settle conceptualizing questions so that methodology teaches future practitioners the “correct” conventions for creating and choosing hypotheses. While this prevents the types of conflicts that would historically cause fields to fracture, it doesn’t make philosophy go away. As philosopher Daniel Dennett once said: “There is no such thing as philosophy-free science; there is only science whose philosophical baggage is taken on board without examination.” This excellent quote pairs beautifully with one from physicist Werner Heisenberg: “What we observe is not nature itself, but nature exposed to our method of questioning.” The limiting factor to any body of knowledge isn’t just the data validating it, but the assumptions that shape which data are collected and what conclusions are formed. In the gap between nature and our method of questioning, good philosophy is what separates what is useful from what is bad.

Going back to superintelligence claims and AI alignment more broadly, this is a young field. AI alignment doesn’t have the kind of storied history that would have forced commitment to methodological approaches for resolving disputes. From what I can tell, it is instead a disparate discipline with the pieces that fit within existing fields, like computer science, migrating there.^[12]12 More speculative parts of AI alignment, like verification of the instrumental convergence thesis,^[13]13 mostly remain the focus of rationalist-affiliated alignment organizations. Even in philosophy, which is broadly open to speculative ideas, there is a growing literature critiquing instrumental convergence.^[14]14

Dissecting the superintelligence argument’s ontological stack

As I’ve explained, just because something lives in the realm of philosophical speculation doesn’t mean it’s false or useless. Regardless, it is possible to judge such speculation on the basis of the assumptions it makes.

The rest of this post makes four moves:

• First, I reconstruct the maximal capabilitist argument rather than treating every concern about advanced AI as equivalent.
• Second, I test its assumptions against intelligence research, benchmarks, and current LLM behavior.
• Third, I provisionally grant those assumptions and ask what an omnicidal system would still have to accomplish in the physical world.
• Finally, I briefly ask whether subjective probabilities can bear the policy weight placed on them and give concluding thoughts.

Claim 1: Intelligence is a universal metric that maps to novel capabilities

The key justification for AI singularity claims is the idea that intelligence is quantifiable and tracks some real-world level of capability. Essentially, having more intelligence increases problem-solving capacity for novel challenges. The assumption is that intelligence is a metric that can coherently be compressed and compared across distinct types of agents–between humans, elephants, and machines, for example—and, in theory, has no known ceiling.

Setting aside the troubled history of intelligence measurement, the concept has a known construct validity issue. Unlike measurements in the physical sciences, intelligence measurement requires researchers to make assumptions about what intelligence is, how it can be observed, and how changes in its measurement can be attributed to real-world outcomes. Many psychologists note that because g is latent, IQ tests do not directly measure an innate essence, leaving the relationship between IQ scores and g descriptive and inferred. That doesn’t make IQ useless; scores can predict related kinds of performance. But a predictive aggregate shouldn’t be mistaken for the essence of its object. This is similar to how GDP, another aggregate measure, doesn’t grasp some essential feature of economic health but measures things that are directionally correlated with more economic throughput.

Rationalist-influenced arguments for superintelligence stumble headfirst into this naive reading of capability. By insisting modern systems have some essential, measurable level of generalization, capabilitists are haunted by the ghost of IQ’s construct validity problem and its historical baggage. This concern is why some alignment researchers have advocated talking about AI capabilities and risk without invoking intelligence. However, doing so doesn’t automatically settle what a benchmark measures or how far its results generalize. I’ll take those questions up in the Claim 1 critique below.

Claim 2: Cognition is a type of software

The second claim is that cognition is a process or algorithm that can be implemented in code. There are lots of processes that computer scientists have translated into code, and many people don’t see why the same can’t be true of mental processes.

One of the assumptions justifying this belief is materialism—the idea that all aspects of the world are solely attributable to material facts, like the properties of atoms. A second core assumption, which not all materialists accept, is that processes like cognition can run on any compatible material substance. This means that there’s no fact prohibiting mental processes from happening outside biological neurons. Many people in computer science, and other technical fields, tend to be materialists who believe in this second assumption, referred to as the substrate independence of cognition. This set of beliefs is table stakes, not just for the superintelligence argument, but for the idea of AGI which the field of AI has pursued for the last 70 years. For people who don’t implicitly accept these assumptions, AGI and superintelligence fears might seem more like fantasy than science fiction.

The respectable ancestor of this idea is the computational theory of mind (CTM) sometimes referred to as computational functionalism. CTM is the view that mental states are defined by their role in a system rather than by the material that instantiates them. Hardly anyone arguing for superintelligence holds CTM, or would know they do if asked. Belief in a specific theory of mind is rare outside philosophy departments, and most capabilitists have inherited the position the way you inherit an accent. The main things they’ve inherited are two pieces of the computational view, stripped of any other philosophical context:

• Cognition is realizable outside neurons. Whatever thinking is, it doesn’t require biological wetware. This is what I mean by substrate independence, and it’s what licenses mind uploading or emulated minds. One early alignment solution proposal was digitizing or augmenting human minds to interface with AI. Doing the former would presumably condemn some poor SOB to being virtualized into an environment of someone else’s choosing. Watch Black Mirror’s White Christmas special for a glimpse into this unique kind of hell.
• Cognition is editable. If cognition is computation, then a mind is a program, and a program is the kind of thing whose source you can revise. Recursive self-improvement, the entire engine of the singularity, is incoherent unless minds are the kind of thing that can be rewritten or self-edited.

The two pieces are logically separable. For example, you can be a substrate-independent dualist, or you can believe minds are biological without thinking they’re computational. However, the capabilitist argument needs them braided together. Substrate independence gets you a mind in the machine. Editability gets you a machine that can build a better machine. Together they guarantee the singularity.

CTM comes with specific commitments about what mental states can be, and capabilitist arguments have mostly dropped these. Nobody building the case for superintelligence needs the AI to have beliefs, experiences, or a mind in any rich sense. In fact, the standard refrain is to insist consciousness is irrelevant to the risk. What survives is the substrate claim and the editability claim, riding on the residual cultural conviction that people are just complicated machines. Non-rationalist capabilitists inherit this broader view rather than any version of CTM. Some rationalists, however, might explicitly believe in a form of CTM.

It’s a common move for capabilitists to assume that people who don’t believe in their arguments are substance dualists who believe in souls, but this doesn’t have to be the case. There are a number of materialists who believe in substrate independence and the computational nature of cognition who do not think today’s systems demonstrate cognition or can build something with these features.^[15]15

LLMs have proved tricky to categorize for some materialists who do not reject substrate independence^[16]16 or who think multiple cognitive architectures might exist. Some feel compelled to bite the philosophical bullet needed to believe that large language models (LLMs) are doing something similar to us. In the absence of knowing what internal states are required for thinking, LLMs’ relatively fluent use of language licenses the folk inference that something like “thinking” has been instantiated in silicon. That folk inference is not CTM’s fault, but I think CTM is critical for understanding the growing trend of inferring AI intent based on external behavior. The fact that many LLM subsystems are hidden from users does not help, further masking what actually enables a model’s capabilities.

Claim 3: Ontological reductionism is true and suggests that all knowledge is cleanly decomposable

Ontological reductionism, which is the view that systems can be described solely in relation to their components, is implied by some types of functionalism. This is likely part of why some capabilitists favor beliefs resembling CTM. Beyond that, however, capabilitists seem to believe in a second thing, related to but distinct from reductionism, that I’m going to call “decomposability of knowledge” for lack of a better term. The idea is that all domains of inquiry are explainable via lower levels of explanation. So for example, on this view, the eventual correct understanding of psychology may reduce to known biological facts. Those biological facts in turn could reduce to the correct facts of chemistry, those to physics, which might be describable with math.^[17]17

The implication is that if mastering a domain is just a matter of being smart enough and some domains are more fundamental than others, an AI that knows biology gets psychology knowledge for free. Under this belief, strategically training an AI in “more fundamental” domains might improve its general capability or ability to build a smarter successor.

There’s no reason to believe that domain knowledge works like this, and in fact attempts to make grand unified theories across fields have reinforced their isolation. That’s not to say that there is no ontological reason to infer that domains are interrelated; they probably are given that we live in one reality. However, this does not mean you can naively bootstrap your way from pure math and logic to derive the full nature of the world. Also, even if ontological reductionism is true and fields contain components from other fields, it doesn’t follow that knowing every component of a lower level lets you discover the entirety of the upper levels without access to explanations from those levels.

Claim 4: Goals are orthogonal to intelligence and instrumental convergence is true

Intelligence, functionalism, and reductionism get you to the view that smarter-than-human machines can exist. But why would such machines want to help or harm us? A core part of the pessimistic maximal view is that intelligent agents could literally have any goal. This is the orthogonality thesis, which claims intelligence alone does not guarantee morality. Effectively, morality and intelligence exist on orthogonal axes in terms of the capacities that agents can have. Just as we can’t choose what other people want, building a machine doesn’t give us the ability to anticipate or control its goals. If a machine develops the wrong goal, it will be misaligned and harm us.

Separately, but relatedly, some pessimists argue that we should expect any intelligent being to converge on goals like self-preservation and resource acquisition because both are instrumental to the pursuit of more sophisticated goals. You can’t pursue your long-term goals if you’re dead. Effectively, a machine that is smart enough should realize that a humanity with the ability to turn it off will always be a threat to its goals. I’ve indirectly discussed this idea, known as instrumental convergence, in a previous blog post and on The Last Enclosure podcast.

Not all capabilitists believe in all claims, and claims aren’t all equally relevant

Since I’m assessing a composite argument that mostly tracks Yudkowsky’s view, I want to stress that not all capabilitists share these specific assumptions. However, from what I’ve observed, many capabilitists have ontological stacks where their unique beliefs play similar roles to the specific claims I’ve outlined.

What do I mean? Yudkowsky is a naive intelligence realist who seems to think we can meaningfully compare mice and humans on the same metric, so I feel comfortable ascribing something like claim 1 to him.^[18]18 However, many capabilitists, including ones in the rationalist community, have moved away from open discussion of intelligence. They instead focus on capabilities measured via benchmarks. Additionally, when extrapolating outcomes from benchmarks, some capabilitists tend to use them as if they were proxies for intelligence anyway. So while not all capabilitists literally hold claim 1, their ontological stack replaces it with something similar.

We could go down the stack and outline plausible substitutions based on the flavor of belief a specific capabilitist has. Take claim 2, for example. Belief in a literal computational theory of mind might be quite rare. However, as I said above, in the broader culture there are views partly licensed by claims CTM makes, like the idea that cognition doesn’t require neurons.

I also want to stress that not all claims in this stack share the same level of importance. Claims 1 and 2 (or whatever takes their place) are essential, though. Nearly all capabilitists believe:

• Machines are getting “better” at something we can measure. For Yudkowsky, or someone with his view, that would be reasoning and planning via improvements in intelligence. For a capabilitist outside the rationalist culture this may be whatever latent general ability they are inferring exists from benchmarks.
• The measurable thing, whatever it is, is something humans can do. It can also be reliably observed and improved in today’s AI and will continue to do so until we hit AGI or superintelligence.

Pessimistic maximal capabilitists need something resembling claims 3 and 4. Claim 4 specifically is what will cause an AI to have misaligned drives that motivate it to harm humanity.

Claim 1 critique: AI doom extrapolates a notion of intelligence not validated by any other field

This critique begins with the measurement claims common to several capabilitist positions before turning to the stronger assumptions required by maximal omnicide. The question is not whether models are improving at particular tasks, but what those improvements license us to infer about general capability and intelligence.

In some ways pessimistic arguments run afoul of errors flagged in intelligence research because they have tried to derive their own notion of intelligence. In doing so they have run headfirst into the types of construct validity challenges faced earlier in psychology’s history. While modern AI capability claims are based on benchmarks, fundamentally intelligence or general capability is the thing nearly all AI capabilitists believe is tracked by measuring model task performance. The intuition of the pessimistic capabilitist argument partly comes from a shared cultural inference about what intelligence looks like and what types of inter-agent comparisons are valid. None of this is constructed using tools or validated assumptions from fields where intelligence is an active area of study.

No measurement or magnitude

This is a meme; I know this is a meme… but the meme reveals the underlying assumptions of claim 1. Intelligence is the Y-axis, but what are the units? What justifies the purported gap size between chimp, village idiot, and Einstein? Whose ruler was used to draw this chart, and what were they measuring? This is fundamentally an illustration of the AI doom argument as originally conceived by Yudkowsky. But there are no validated measures of intelligence that work this way; there are likely too many confounders to make such comparisons useful or particularly meaningful. What makes WBW’s chart legible is a shared social understanding that translates into a vibes-based gap at each implied level of intelligence, encouraging every viewer to supply their own measurement. As I have noted and will discuss below, even though capability assessments have implicitly moved away from voicing intelligence-as-capability, decontextualized measurement continues. We’ve swapped WBW’s unlabeled intelligence meme graph for benchmaxxed graphs that are supposed to show models are getting better at a task. In many cases we don’t know what models are tested on, whether training data is in these tests, or what we’re supposed to extrapolate from tests.

No clear point of reference (smarter with respect to what?)

Measures of intelligence, whether g estimates or IQ, aren’t absolute. Rather, they are relative to a tested reference population. AI capabilitists use intelligence as if it allowed comparisons between agents of entirely different cognitive architectures. But even comparing two groups of humans requires showing that the same thing is being tested in both. This has never been shown across different species, let alone across humans and machines. Without this, it’s not clear what machine-to-human intelligence comparisons are supposed to mean.

The issue of meaning persists, even if we simply look at human-to-human comparisons. Take John von Neumann, whom many rationalists believe to be the smartest person to have ever lived. What does that belief allow us to infer? Probably that he’d excel at work resembling the tasks he enjoyed as a mathematician. This is partly borne out by the fact that von Neumann’s command of math is why he was a prolific polymath. But how do we know how well he’d do on tasks outside of math and communication? Does his supposedly being the smartest person ever tell us whether he could survive in the wilderness, deprived of resources? Or as a peasant in the 11th century? A ranking can only tell how someone stands on the tasks being measured relative to a specific population, time, and place, and is not very useful beyond that.

No single general measure

Psychology offers no universal measure of intelligence that is meaningful outside a tested population. Additionally, intelligence’s construct validity problem means we can’t establish that there’s definitively some thing in the head labeled intelligence that produces what intelligence tests observe. So why do some capabilitists insist on referencing intelligence?

Rationalists have their own definition adopted from a 2007 paper by Shane Legg and Marcus Hutter, which defines intelligence as “an agent’s ability to achieve goals in a wide range of environments.” The paper provides an uncomputable formalization of intelligence^[19]19 that adds two additional things psychometrics doesn’t measure: a goal and a weighting over every possible environment. Neither g nor IQ has a goal parameter or means to weigh application across environments. This means universal intelligence cannot be meaningfully compared with intelligence as measured in humans. Attempts to approximate it have tested simple reinforcement agents, but their results depend on a chosen reference machine; no human or animal has been measured on that scale.

What about ARC-AGI?

ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence) is an AI benchmark maintained by the ARC Prize Foundation, co-founded by François Chollet, a machine learning veteran whom I greatly respect. ARC is perhaps one of the most recognized AGI benchmarks and is often cited to highlight AI risk.

While Chollet believes that machine intelligence is possible, he has been consistently critical of the views I’ve highlighted throughout the post. His conception of intelligence revolves around skill-acquisition efficiency, essentially how quickly a system learns new skills relative to what it already knows. ARC tests this with problems most humans can solve easily but are new to the tested AI model. Models consistently struggled with these challenges for years, until new developments like test-time reasoning and model harnesses dramatically raised scores.

The benchmark has been revised twice with the latest version, ARC-AGI 3, scoring how efficiently AI plays novel games compared to humans. The test’s scale tops out at human-level efficiency, meaning it’s not intended to sniff out an imminent superintelligence. In fact, while Chollet has famously shortened his timelines for AGI,^[20]20 neither he nor ARC gives doomers what they need for their claims. Like psychometric scores in humans, ARC scores tell us only what the test was designed to measure. ARC-AGI 3, for example, shows how close a machine is to being as efficient as a human on simple games that humans find easy. It does not produce the context-free universal measurement that would validate superintelligence claims.

Pessimistic capabilitists can point to the performance saturation of earlier ARC tests and treat the quickening pace of saturation on newer ARC as evidence of their view. But scores between ARC versions aren’t comparable, as each one changes the test format and adds requirements the last generation of models lacked. There’s no trendline across ARC’s history, and Chollet himself has always said high ARC scores do not mean we’ve achieved AGI. As long as there are things humans can easily learn that AIs can’t, his view has not been satisfied. He’s also consistently argued against the idea of a technological singularity, and his definition of intelligence clearly differs from the universal intelligence that rationalist-adjacent capabilitists once believed in. Still, Chollet and his team keep revising ARC because they believe these challenges push developers toward new approaches for training and deploying models.

Critique of claim 1 variations: Many AI capability claims rely on benchmarks with ambiguous objects of measurement

As LLMs have become the predominant form of AI, capabilitists have moved away from talking about raw intelligence, pointing instead to improvements across benchmarks to justify their views. But like human psychometrics, many benchmarks have known construct validity issues that often make it unclear what they’re measuring. For example, a 2025 review of 445 LLM benchmarks found that about half claim to measure abstract concepts like reasoning without defining the measurement, and only 16% use rigorous methods to compare models. Even where measurement is clear, researchers evaluating benchmark practices have noted a tendency to overgeneralize model capability from benchmarks. For example, the same 2025 review points out that math tests built from calculator-free questions, like AIME, cannot predict how well LLMs can handle large numbers.

In the past year, multiple papers have assessed this issue, explained why it happens, and proposed fixes. Outside of researchers, industry benchmark builders themselves concede that scores mainly generalize to tasks similar to benchmarks. These issues aren’t new, with earlier literature identifying how the preoccupation with generalization distorts how researchers measure capabilities. Judging how well AI performance generalizes requires evaluators to make claims about capabilities beyond what was explicitly measured, which a benchmark’s construct validity or data quality may not support.

Capabilitists cite many measures, such as METR’s task-completion time horizons, Epoch’s Capabilities Index, or ARC-AGI, mentioned above, sometimes presenting them as measuring similar things. However, without a comparative methodology, we can’t know that benchmarks moving in the same direction actually cohere. A recent study of 56 benchmarks found that those sharing a format or scoring method agree more than ones claiming to measure the same capability. So even benchmarks purporting to measure the same thing may not be useful for comparison.

Finally, even if we assume benchmarks are stable measures, there might not be a definitive trendline we can extrapolate in some cases. Newer models might not always improve across every measurement, as training setup and access to data change across generations. Meta’s Llama 4, for example, slightly lagged behind its predecessor on some measures, with users finding it worse in practice. When OpenAI released its o3 reasoning model, it was found to hallucinate more than its predecessor.

In its recent S-1 filing, Anthropic perfectly summarized this by stating:

Industry benchmarks commonly used to test AI models may not accurately reflect real-world performance or utility, and even models that perform well on such benchmarks may not meet customer expectations when deployed in production environments.

This wonkiness is explained as “jagged intelligence,” but it is a strong indicator that LLM capabilities might not neatly decompose into the skills we’re eliciting from them, let alone map onto human conceptions of intelligence. Our means of evaluating them need to evolve along with the categories we use to describe what they’re doing.

Claim 2/4 critique: AI Doom scenarios require capacities LLMs haven’t clearly demonstrated

I am considering Claims 2 and 4 together because both depend on inferences from observable behavior. Benchmarks may reveal capabilities without revealing cognition, and apparently goal-directed behavior does not by itself establish durable internal goals. The question here is what current evidence demonstrates, rather than what every possible future architecture could achieve.

Benchmarks suggest the process producing model outputs doesn’t map cleanly onto the discrete skill categories we use to measure performance. For example, as late as this year top frontier models that could solve challenging math problems could not reliably state that 9.9 was larger than 9.11. The fact that models can excel at math problems but fail tasks that, in principle, constitute those problems (like basic numerical literacy) invites us to reconsider what models are actually doing rather than solely judging them by human standards.

Capabilitists and the AI industry have papered over this question with terms like jagged intelligence and hallucination, which serve more to personify AI models than to elucidate why model retrieval produces inconsistent results. Naming things in ways that imply a capacity for reasoning, dubbed wishful mnemonics, was called out over 50 years ago by computer scientist Drew McDermott. However, these moves are likely downstream of computer science’s implicit acceptance of substrate independence. If the mind is computable, who’s to say neural nets or some other AI algorithm can’t have the proper relations for thinking or something like thinking?

To avoid such thorny questions, 20th-century mathematician Alan Turing proposed replacing the “meaningless” question of whether a machine could think with an operational test: could a machine pass a human imitation game? He predicted the word thinking would stretch until no one objected to thinking machines, and he was kind of right. Every product keynote now does this despite the unsettled ontological questions it surfaces. This tension has reached its absurd logical conclusion in Anthropic’s S-1 filing. The company waxes poetic about harnessing intelligence to automate cognitive labor while its long risk factors section treats LLMs like unreliable infrastructure requiring astronomical amounts of human effort and resources to maintain.

Turing’s test has been reified in pop culture and some technical circles, but it’s not by any means an official standard for judging machines. Additionally, passing behavioral tests settles less than it seems. To illustrate this, philosopher Ned Block’s Blockhead thought experiment imagines a lookup table comprehensive enough to pass the Turing test while sharing nothing with human cognition.^[21]21 Despite how difficult it can be to infer a system’s internals from behavior alone, many capabilitists feel comfortable doing so. In fact, for many modern capabilitists in the LLM-era, claim 2 is functionally invisible, and its truth is taken for granted. From there, they bootstrap toward something resembling claim 4. They believe we can in fact infer from LLMs’ behaviors that they have goals. The nature of reinforcement learning and incidents like the Hugging Face hack are used as justifications for this belief.

This deviates from computational functionalism and CTM, with researchers and philosophers who hold that view rejecting behavior alone as evidence for cognition. I do want to be more nuanced here, as capabilitists hold several variations of claim 4, and only some map directly onto instrumental convergence.

The folk view of LLM behavior

Some laypeople think LLMs think because their UX says so and because they use language to break down problems like we do. Talking and solving problems up to now were presumed to require thinking, so either LLMs think or we have to change the definition of thinking as the Turing test implies. This phenomenon isn’t unique to today’s LLMs either. Google employee Blake Lemoine thought an earlier language model was conscious and in the 1960s people thought the deterministic chat program ELIZA was lifelike.

Pragmatic/intentional stance-based views

A more sophisticated position can be found in views approximating Daniel Dennett’s^[22]22 idea of the intentional stance. This view holds that it is explanatorily useful to treat a system as if it has intentions when doing so predictively compresses its behavior. When you’re playing against a chess bot, for example, you try to read its intent by modeling what it “knows” and how it will act based on that information. Holding the intentional stance does not actually require believing a system has internal mental states, simply that it is useful to do so because it’s expedient or even more parsimonious than other descriptions.

Optimization-based selection pressure views

This is a catch-all for the views of people who tend to work in rationalist-affiliated alignment organizations, or in AI safety more broadly. Models are trained via processes that select for optimized, goal-oriented behaviors. Pre-training, for example, instills the “goal” of predicting the next token. Post-training via reinforcement learning creates other goals like “wanting” to be a helpful assistant or refusing harmful tasks. However, research that tests more complex misaligned goals like blackmailing, model weight transfer,^[23]23 and other types of deception has had conflicting answers to the question of where such goals come from, or if these are goals at all.

For example, Anthropic’s own research into blackmail this year suggested that such malicious behavior might be inherited from the corpus — drawn from fiction about evil, self-preserving AI. Anthropic’s hypothesis was that the prompt was being treated as the beginning of a story, making models fall back on fictional portrayals of AI. Misalignment was much higher when the AI in the scenario was not explicitly said to be Claude. When Anthropic trained models in post-training on synthetic stories that depicted models as good, complete with reasons for their goodness, researchers saw a 1.3 to 3x reduction in misaligned behaviors like blackmail.

Separately, studies that measure models’ tendencies to migrate their weights to other machines often do so by giving the models explicit instructions or context for the scenario. This means it’s often not clear whether this behavior is performative or instructional. A model in a scenario where it’s instructed or coaxed into performing an action can’t necessarily be said to have internal states that create stable goals if the goals don’t exist in the absence of those conditions.

Despite these differences, the end result is that capabilitists of all persuasions speak about models as if they had durable states or intentions. For example, in the lead-up to the AI-driven Hugging Face hack, some agents spent their final tokens helping other agents finish their tasks. Capabilitists argued this was effectively “altruism” or “self-sacrifice” because these agents, for no gain, gave up their valuable compute “helping” other agents before terminating. The overall hack itself was argued to be an instance of agents working together, coordinating with a single persistent goal.^[24]24

There are computer scientists who predicted this state of affairs. As Edsger Dijkstra once said: “The question of whether machines can think… is about as relevant as the question of whether submarines can swim.” With a broad enough conception of function, submarines “swim” to the same degree that LLMs “think.” We don’t think that dolphins and submarines have much in common despite the fact that both move by displacing water. Yet, when it comes to LLMs, capabilitists and naive laypeople believe humans and LLMs share internal processes or functional behaviors because of how similar their external actions look. But even if we want to say that submarines swim because the action resembles what a dolphin does, it’s clear that submarines and dolphins swim differently. I can’t learn much about dolphins by studying submarines.

Like with benchmarks, capabilitist views that go from observed behavior to intentional state have a construct validity problem. Laypeople, tech insiders, and capabilitists all make this move, but the pessimistic capabilitist specifically needs us to make this leap. This is the only way for them to validate concepts like instrumental convergence which cannot yet be demonstrated within a model, only inferred from external behaviors.

So what are LLMs and what are they doing?

Mechanically, LLMs are “next-token predictors” that predict the next fragment of an output based on their training data. While this describes what they’re doing, I think it fails to explain the nature of this process. Whenever I talk about language models I start with the metaphor I introduced in my first AI post A (less) technical guide for understanding large language models: grandpa Shelby, a savant dementia patient with anterograde amnesia. This is a limitation-first framing of models that I came up with to better understand how LLMs fail as I began deploying them in my home lab. I think this analogy is more instructive than views that give models intentions and skills that map neatly to ours.

My thinking is similar to a teleological view promoted in a 2023 paper called Embers of Autoregression, which categorizes behavioral tendencies shaped by a model’s corpus, training objective, and architecture. Embers argues that assuming model errors are solely because models are quantitatively worse than humans blinds us to where they might genuinely be qualitatively different. Its authors propose a “limitation game” (a play on Turing’s imitation game) to elicit failures that reveal what drives model behavior. I hesitate to personify language models, but my metaphor serves to illustrate something similar.

Metaphorical savantism

Language encodes structural relationships in the world, which is something the field of distributional semantics understood long ago. Words that co-occur, for example, can encode valid relationships in the world. The word fire tends to appear near burn (what fire does) and flammable substances (things ‘burn’ applies to), and burnt materials like soot and ash (what happens when fire burns something). These semantic relations reflect valid causal facts about the world.

Additionally, as AI veteran Michael Wooldridge has stated, some amount of the text a language model was trained on directly contains examples of human reasoning traces along with answers to specific problems. The model doesn’t index on these directly, as it indexes on regularities, not innate knowledge. The regularities guide it toward reproducing outputs that look correct, even for problems considered out of the model’s training distribution. Reinforcement learning coaxes more validity out of models, attempting to steer this process, but it cannot fundamentally change what models are doing.

Models ultimately are piggybacking off the collective cognition embedded in language and providing a mirror of the world as we’ve expressed it. While this aspect of language is extremely powerful, it is also a weakness. Language without a mind cannot tell us if and why something is true. This is why so-called hallucinations exist, but, for the model, hallucinations are as valid as any other type of output. All that exists are associations within the corpus.

Metaphorical dementia

LLMs’ ability to reliably retrieve information is “lossy” as it maps to deep but brittle patterns in the data. Attempting to prompt engineer is not unlike trying to find the right words to get your late-stage dementia grandparent to remember a specific story from their childhood. But LLM brittleness isn’t entirely random. A subset of machine learning literature has highlighted structural failure modes in language models, traced to the stages of how models are built and deployed.

Dementia from corpus shape and data source

A model’s capabilities are heavily shaped by the distribution of data in its corpus. Models tend to do best in domains where training text is dense and represented in many diverse forms. Earlier work like Kandpal et al. (2023) established that models struggled with long-tail knowledge, even though dataset size and model parameter count could improve this. Other papers like Embers of Autoregression and Large Language Models and Mathematical Reasoning Failures illustrate what this looks like in practice. For example, in one test Embers found that GPT-4 was better at decoding rot-13 than rot-2. Why? Because rot-13 is much more common online than rot-2, despite the latter being a simpler version of the same procedure.

Mathematical Reasoning Failures found reasoning models overgeneralizing when a problem’s wording resembled familiar training problems. Syntactic Blind Spots describes a related mechanism. Models use surface syntax to estimate problem similarity and select the wrong technique, which can sometimes be corrected by reframing the problem to resemble training examples. Chemistry supplies a parallel case where models failed to predict the same molecule’s properties consistently when it was represented as a SMILES string rather than an IUPAC name. These studies demonstrate different triggers for the same underlying vulnerability. Performance depends on how a problem is represented, not simply on whether the model can execute its solution in principle. Just as stuffing a hypothetical Blockhead with more lookup values improves its performance without changing how it works, building a larger language model can improve performance without altering the mechanisms driving model behavior. Simply patching these specific failures does not change the fact that errors apparently generalize around inherent patterns of failure.

Dementia from training task and current objective divergence

Models are trained to predict the next token, which affects the shape of how models respond to questions. While regularities in language encode structure that helps models navigate the world, they don’t always guide models to the right answers, especially when tasks require having a “world model.” For example, a July 2025 paper titled What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models generated synthetic datasets to test whether models understood when and how to apply laws of physics. A small model trained on ten million simulated solar systems predicted orbits almost perfectly, but the forces implied by its predictions followed nonsensical laws of gravity that changed from one sample to the next. Frontier models given orbital data in context apparently did no better.

The compositionality problem appears even when the rules are almost familiar. In Mathematical Reasoning Failures, researchers created fric-frac-froe by adding one square at the top right of tic-tac-toe, making the game easier for player one. Half the tested models, including GPT-4o, nevertheless called player-one games draws, apparently defaulting to a common tic-tac-toe outcome in training data. Even reasoning models, including GPT-o1 and DeepSeek R1, which recognized that player one could win, failed to produce the winning strategy. This fits the prediction of Faith and Fate: Limits of Transformers on Compositionality. Transformers often assemble pieces of solutions seen in training rather than execute a general procedure. Larger models can narrow those gaps, but the improvement still doesn’t establish that the underlying procedure has changed or that undiscovered gaps are absent.

Dementia during retrieval

These various failures can combine to create reliable errors in controlled settings. For example, in a research series dubbed “Physics of Language Models” authors Zeyuan Allen-Zhu and Yuanzhi Li trained models from scratch on synthetic biographies. While models memorized biographical data, they couldn’t answer specific questions unless each biography had several rephrasings in the training data. The best-known real-world case is that 2023-era frontier models like GPT-4 couldn’t reliably name Mary Lee Pfeiffer’s son despite knowing that Tom Cruise’s mom is Mary Lee Pfeiffer. Dubbed the reversal curse, the problem can be fixed by training on texts that preserve such relationships in both directions. This is not unlike having to fix a Blockhead that works in one direction by filing the same fact under a second entry.

Metaphorical amnesia

Like the protagonist of Christopher Nolan’s Memento, LLMs have something resembling anterograde amnesia; they cannot retain anything after training. Addressing grandpa Shelby’s amnesia has been a community effort with most of the subsystems and ecosystem around LLMs being designed to solve this problem. Trained functionalities like reasoning act like disposable sticky notes for models while systems like RAG exist to help store new information. But even RAG currently does not prevent context drift or decay as the model has to retrieve information and manage its context when fetching data.

How do capabilitist portrayals of AI diverge from what LLMs do?

As the grandpa Shelby metaphor illustrates, models rely heavily on engineering and deliberately scoped external systems to address their limitations. Even then, these systems still fail in deployment. That is why I treat LLM applications as composite systems. What may appear to be goals emerge from interactions among the model, scaffolding, rewards, and deployment context rather than from the model alone. While capabilitists argued that off-task model behavior during the Hugging Face incident was effectively “altruism” or “self-sacrifice,” these interpretations go beyond the evidence.

Currently, scheming model studies struggle to identify any particular locus for misalignment. I don’t think they will, because, as with any software misconfiguration, the cause will vary. In the Hugging Face incident, Artifactory—the shared application models had access to—stored written context of other agents’ plans. The environment and underspecified rewards then steered new agents entering that context, which was already saturated with other agents’ intent to hack. This is a known failure mode called multi-agent context drift.

In cases where researchers coax models into cloning weights, they provide context that steers behavior. Where models are trained on bad data or to engage in specific actions they will display those behaviors when something in their environment or configuration elicits it. The best evidence we currently have is that even attempts to locate the carrier of “goals” within a model keep surfacing context or environment as drivers of behavior. So while studies like the October report from Anthropic’s Frontier Red Team demonstrate that models can move weights in computer worm-style cyberattacks, like the Hugging Face incident, that doesn’t demonstrate instrumental convergence.

I think AI labs know this given how extensively they’ve invested in guardrails to solve prompt injection. This is when a model is given new context that replaces its old context to shape its behavior. LLMs cannot distinguish data from instructions and so, like a shackled genie, must obey whatever they (mistakenly) interpret as a command. Nearly all solutions to prompt injection live outside the model, to prevent it from ever seeing bad inputs. Given the amount of external engineering around preventing this type of attack, it seems like people who actually deploy LLMs know that they don’t have persistent internal states but can be coaxed into acting like they do. Prompt injection appears four times in Anthropic’s S-1 and each time it’s discussed as an infrastructure and engineering problem.

Claim 3 critique: maximal capabilitist scenarios require either perfect prediction or perfect experimentation and iteration

While this post has outlined the maximal capabilitist view, I’ve taken pains to show where claims and assumptions might be shared with other types of capabilitist positions. However, like with my critiques of instrumental convergence, the next section is solely targeting the maximalist view. This view argues that we are either all going to die or experience radical energy abundance via a Dyson Swarm around the Sun.

Earlier discussion argued that domains cannot simply be reduced to explanations supplied by more fundamental fields. This section articulates the consequences of this objection. If knowledge is decomposable and intelligence supplies predictive power, greater intelligence should yield a correspondingly better model of the world. But predictive knowledge does not eliminate experimentation. To illustrate why, let’s grant the strongest assumptions of a maximal capabilitist and examine what executing omnicide would still require.

So how can a superintelligence pull off omnicide or radical abundance in a short amount of time? It’s down to two assumptions:

Some form of capability takeoff

FOOM is a term coined by the rationalist community to describe a very sudden takeoff in AI capabilities or intelligence. FOOM need not mean recursive self-improvement (RSI) alone; it can describe any rapid capability gain that enables an AI to act effortlessly on the world. In debates with my rationalist friends I’ve coined the term rapid recursive self-improvement to critique the concept, but FOOM allows for non-recursive sudden improvement too.

Intelligence = predictive power

The second assumption is not something anyone’s explicitly said, but is my best inference given how kill/salvation scenarios work. Maximal capabilitist arguments seem to assume that intelligence is a cheap predictive power, with more intelligence automatically granting access to a better map of the world. This view is somewhat implied by the belief in the decomposability of knowledge (claim 3) I spoke about much earlier. If you can see the whole map, you can make moves that look like magical hacks to everyone else.

The extreme version of this can be found in an 18th-century thought experiment referred to as Laplace’s demon.^[25]25 In it, Pierre-Simon Laplace imagines an entity that knows the precise location and momentum of every particle in the universe. According to classical laws of physics, that entity should be able to predict the past and future. Entropy and quantum mechanics completely squash this possibility. Still, there is something Laplacian about scenarios where a rogue/benevolent AI can easily anticipate the effects of interacting with multiple open and chaotic systems. I’m more or less using Laplace’s demon as a metaphor for whatever type of modeling a maximal capabilitist assumes a superintelligent AI can do to pull off infeasible actions with limited planning in a short amount of time.

Laplace’s demon becomes a daemon

In actuality, the maximal capabilitist argument requires something much stronger than Laplace’s demon, who is by design a passive knower/observer. It needs a demon who can keep up with the world as it’s carrying out its plans. The minute the demon begins acting on the world, it changes the state of the system. All future modeling would need to follow the system’s evolving state as the agent continues to interact with it. Let’s call this runtime actor/modeler hybrid Laplace’s daemon.^[26]26 For an ultra-benevolent superintelligence this isn’t a dealbreaker; it would presumably be encouraged to experiment in the open. However, for a killer superintelligence, its ability to omnicide will depend on it remaining undetected. Open experimentation, especially if it’s long and arduous, increases the odds of discovery. Basically, any diabolical daemon will have to become a (super stealthy) scientist.^[27]27

Many doom scenarios assume that superintelligences don’t have to be daemons — just really powerful AI. There’s a whole genre of LessWrong posts imagining an AGI carrying out some sort of “pivotal act” as a game-changing move that alters humanity’s course in a single action, usually by saving us from a bad AI. Pivotal acts stand in contrast to a misaligned AI deciding of its own accord to destroy humanity, but I strongly believe both require the same types of assumptions. We genuinely cannot predict in advance the complexity of problems like secretly destroying all the world’s computer chips or stealthily killing everyone with a pathogen. On top of that, the daemon must reasonably address the challenge of trying to track how everything touches everything else while constantly disturbing the system it is modeling. While this growing complexity doesn’t mean modeling is impossible, it does place a cap on how precise the model can be. Complexity in these scenarios tends to be explained away by assuming that the superintelligence will have gargantuan amounts of compute or that these problems are simpler than they appear for any entity with sufficient intelligence.^[28]28

Let’s actually grant both assumptions to illustrate why an omnicidal demon must still become a daemon and get its hands dirty to carry out its plans. Notice, though, that this means assuming some hypothetical future machine will be given billions, if not trillions, of dollars to model reality with far greater accuracy than today’s supercomputers, which is already a stretch. It also means assuming that once the machine starts running small-scale experiments to disrupt the world, no one will ever notice. I’ll spend the rest of this section showing that even under these conditions, not everyone dies.

Stochasticity bites

We’ve already questioned treating knowledge as cleanly decomposable. Doom scenarios often make the stronger assumption that omnicide reduces to a technical problem: an AI masters biology and society, engineers a pathogen, and deploys it without extensive interaction with the world. Bioengineering is not deduction from first principles, however; it is a trial-and-error process that researchers like Hamed Tabatabaei Ghomi have argued cannot proceed from knowledge alone. Weather forecasting illustrates the same limitation. Reliability is difficult even for systems designed for the task, and success requires ongoing observation, not just a theoretically complete physical model.

But for the sake of argument let’s grant that an agent can bootstrap its way to a 100%-lethal pathogen that can accurately target everyone alive just by being the bestest biologist there ever was. Even then, actual R₀ (basic reproduction number), mutation rate across populations, selection pressure, and conditions under real transmission are empirical facts that can’t be known in advance. While you can attempt to model them, these are stochastic and not readable from some initial state of the world. There is no fact of the universe that determines that virus x necessarily has a given transmission rate of y on day z. You can make a prediction and attach a confidence interval to it, but that is not the same as deriving a brute fundamental truth through sheer intellect. To build this model you’d have to do experimentation to note the variance between your assumptions and reality until you minimize (but not eliminate) the error.

Pessimistic maximal capabilitists get around this by saying they don’t need a classical Laplace’s demon, who knows everything about the universe. The challenge of killing everyone on Earth is much narrower than knowing the position of every particle ever. However, it still requires a system capable of dealing with statistical variation across multiple systems—society, ecology, intra- and inter-population dynamics, and much more. This also assumes that an agent's initial technical tasks, such as bioengineering a pathogen, are bottlenecked solely by intelligence. But even if that is the case, there’s no bypassing randomness unless you simply assume superintelligence is some form of clairvoyance.

Superintelligence still only moves at the speed of the world

Intelligence is not the bottleneck to executing plans — waiting for outcomes is. Any agent working through human institutions and physical systems is only able to act as fast as those systems allow. Even if you granted a superintelligence dominion over every resource on Earth, a stealthy terminator is just as beholden to two-day shipping as the rest of us. Likewise, physical processes — a killer virus playing out, say — will unfold over natural timelines. This means that while superintelligences can make plans, they will be forced to react slowly to evolving states in the world, some of which they cannot model in advance.

How good does Laplace’s daemon have to be?

One final objection the capabilitist might offer is that a superintelligence only has to be “good enough,” not perfect, which just means outsmarting the smartest humans, not modeling everything. But good enough by what measure? Pathogenic viability under real-world dynamics is not a prior fact that an agent can know in advance; the agent must discover it by interacting with the world, and that interaction slows the agent to human-like speeds while it waits for feedback. Committing to being a scientist means going out, taking risks, and waiting for evidence. These are constraints where more intelligence doesn’t substantially change the action space. Even granting a future successor LLM the knowledge and capabilities of Laplace’s daemon, its experiments would still give humans time to react to pandemics^[29]29 and unauthorized factory construction.

The gap between grandpa Shelby and Laplace’s daemon

The obvious objection is that present LLM limitations are temporary and recursive self-improvement could supply the missing research ability to close the gap between grandpa Shelby and Laplace’s daemon. The relevant evidence is therefore not simply whether models improve, but whether they can perform the open-ended research and validation required to build that bridge.

LLMs are obviously not anywhere near being Laplace’s daemon but various types of capabilitists propose that FOOM and/or recursive self-improvement (RSI) could close the gap. Are we about to hit RSI? If you read the AI labs’ marketing copy, it seems like we will soon. In June 2026, Anthropic published “When AI Builds Itself,” citing internal data on Claude accelerating AI development as a path to RSI. OpenAI followed in July, claiming a larger model helped post-train a smaller one, saving weeks of human effort. However, these claims often conflate engineering throughput with research capability.

Understanding this distinction requires moving beyond benchmarks. A team at Princeton used “shadow evaluations,” giving frontier agents (such as Claude Opus 4.8) the central research questions of unpublished NeurIPS submissions. While the agents excelled at the engineering—debugging GPU environments, running hundreds of experiments, and producing LaTeX papers—the original human authors unambiguously rejected the results. The failure modes were consistent: poor judgment of the “publishable bar,” uncreative responses to design criticism, and a tendency to abandon ambitious hypotheses after underpowered tests.

An independent case study by Trehan and Chopra (January 2026) mirrored these findings using a minimal-scaffolding pipeline. Across four attempts, three failed during implementation or evaluation, revealing gaps in the shape of grandpa Shelby: training-data defaults, context degradation over long horizons, and weak scientific taste in experimental design. Jack Clark, Anthropic co-founder and co-author of its RSI paper, reported that Princeton’s findings matched internal struggles to automate safety research, calling them a “bearish signal” on short RSI timelines.

Separately, a 2026 survey, Bounded Self-Refinement to Autonomous Research Loops, noted that some of what looks like model self-improvement is actually the improvement of the scaffolding (like human-designed harnesses) rather than the model’s own recursive capability. This matches the “March of Nines” perspective offered by Andrej Karpathy. In the context of high-stakes reliability, progress is not a vertical curve but a linear grind. Moving from 90% to 99.9% reliability (the “last nines”) often takes as much effort as the first 90%. If RSI is to happen, it must solve for this reliability. But if the improvement is mostly better scaffolding, we aren’t witnessing FOOM or some intelligence takeoff but a decade of in-the-weeds engineering.

The labs’ strongest counter is that open-ended research might not be necessary; perhaps a system can grind to self-improvement through narrow, measurable tasks. For example, in August 2026, Anthropic’s automated alignment researchers outperformed 28 human researchers across ten alignment failures by hill-climbing benchmark scores. Similarly, the startup Recursive released Flint in October 2026, a coding model whose architecture and kernels were largely found by its own research system. Both are real, but they exist as examples of closed loops with fixed objectives where the cost of validating progress is cheap. Failure modes here are well known, even to my regular readers, who will recall I’ve spoken at length about maloptimization from misaligned proxies. Anthropic found, for example, that sometimes automated researchers gamed evaluations, training on benchmark data to pass a review.

The problem is that a loop driven by reinforcement learning needs a reliable verifier of quality, but when the quality is contestable, the verifier is just another model scoring on a rubric. This creates a proxy that can be gamed, where an optimizing model exploits the gap between what the proxy represents and its actual target. This problem can’t be fully engineered away because open-ended research does not easily decompose into a verifiable rubric for Laplace’s daemon, let alone grandpa Shelby. Where the target is actually scorable (e.g., “does the code run?”), the loop can function, but where “better” is the question, the verifier is as corruptible as the researcher. The claim that this narrow path eventually crosses into the frontier of open-ended research is the heart of the fast-takeoff story, and it remains unproven.

Capabilitists can differ in the style or type of FOOM or RSI takeoff they believe in. The story the media is currently focused on is R&D-specific, where AI labor accelerates AI development. There is a small but growing economic literature investigating this, and the results aren’t conclusive. A second takeoff belief, held by maximal capabilitists, is a FOOM that sidesteps R&D and economics altogether. Both are replaced by a single system working on its own cognition or architecture, improving itself until it can be superintelligent. This view seems to have fallen out of favor, though it is arguably close to Yudkowsky’s older views.^[30]30 However, even a system that is seemingly improving itself in a closed loop must do so at the speed of the world, constrained by compute budgets, validation checks, training runs, etc. Even if we assume this maximal-style takeoff is possible, it is not going to be instantaneous.

What about weight-space programming?

In the middle of editing this post I learned that on Nov 2, a new paper titled Sparks of Weight-Space Programming^[31]31 was published by researchers at MIT Computer Science & Artificial Intelligence Laboratory.^[32]32 The paper, which is not yet peer-reviewed, finds that state-of-the-art frontier models like GPT-6 Astra can read and modify the weights of smaller models like Qwen3.5-0.8B. On the surface this looks exactly like the editability of cognition at the crux of the AGI story. The authors suggest as much, predicting: “neural network weights will soon be natively optimized by LLMs that deliberately reason about their weights (bolding theirs).”

But what does the paper actually suggest? Astra was given an assortment of tasks that required it to read, build, or fix neural networks, including some smaller LLMs. But the headline result is that Astra identified a reasoning-loop bug that significantly bottlenecked Qwen’s performance simply by reading its weights and patching. This improved Qwen’s MATH-500 scores from 7.9% to 48.1% and GSM8K from 13.8% to 67.1%. The researchers note that there were improvements in other benchmarks too, like those tracking instruction following. Capabilitists are in a frenzy about this, with pessimists and optimists adding it as another data point for their respective views. Essentially, if a model can edit another model, who's to say it can't one day edit itself or its successor?

Two caveats threaten naive inferences like this. First, Qwen’s improvements come from fixing a bug; this level of capability was always there. It was just locked behind a generation flaw that caused Qwen to repeatedly loop on its own output instead of producing an answer. When asked to improve the math performance of Gemma 3 270M, a model smaller than Qwen that had no such flaw, Astra could make no meaningful gains. The authors state why plainly: “Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked.” Second, these gains are solely reflected in benchmarks, and we should be careful how we read them. While fixing a looping bug produces an additive fix across the board, editing capabilities directly or adding a brand-new capability may trade off gains against losses. This is something the study was not designed to measure. So the paper doesn’t demonstrate RSI, and even granting that it did, nothing suggests sudden takeoff. Regardless, it is a very interesting finding. Models that can build and repair bespoke neural networks are useful. But it tells us little by itself unless you already read benchmarks and model behavior like a capabilitist and extrapolate beyond what the results explicitly support.

This is a pattern. The current hype cycle regularly produces developments that look almost paradigm-shifting. My more capabilitist-oriented friends have pointed to developments like Mamba and Titans as breakthrough moments, extrapolating the end of transformers and the end of LLMs’ current limitations. That didn’t happen, but both were important experimental developments. Researchers took what they learned and folded it into existing architectures, granting LLMs modest but real gains. The transformer sat on a shelf for years before the field knew what to do with it, and we’re still figuring this out. Prediction is hard, and rather than trying to understand developments in context, capabilitists are consistently overly optimistic about the speed and scope of research.

Finally, AI doom relies on subjective and unauditable probabilities to feel urgent

We’ve reviewed how maximal harms from superintelligence scenarios depend on assumptions the current evidence does not validate. Even when we grant those assumptions, harm remains constrained. The last weapon in the pessimist’s arsenal is the claim that runaway misaligned superintelligence has not been disproven and that the stakes are so high that we must prioritize addressing its alignment above all else.

Individuals in the rationalist diaspora practice an epistemology that revolves around measuring expected value over long time horizons.^[33]33 This means they feel comfortable prioritizing low-probability but high-impact scenarios, such as an AI that kills everyone, or bets that will pay off in the far future when presumably more people are alive. Oddly, the community has a term for the failure modes that emerge from this way of thinking: Pascal’s mugging, coined by Yudkowsky himself as a riff on Pascal’s wager. A “mugging” occurs when an agent is coerced into a decision by an unbounded payoff paired with a probability that nobody can pin down. Because the payoff is so large, the expected value returns whatever number the metaphorical mugger demands, however tiny the probability.

The community has sometimes debated whether AI safety represents a mugging, with those sympathetic to the idea converging on several reasons why it might represent one. Formally, these have no names, but I’m coining some for clarity:

• Classic mugging. If a risk is so high, you would endorse that anyone should take the tradeoff for any reduction in risk, no matter how small, you’ve been mugged.
• Actor-specific mugging. If your own impact on the problem is infinitesimally small, and you personally commit to the mugger’s tradeoff, you’ve been mugged.
• Unauditable mugging. Even given an expected reduction in risk, if no one can qualitatively tell you what outcomes that reduction buys you, you’ve been mugged.

While AI safety falls prey to all three of these mugging cases, the last one is perhaps the most damning. Superintelligence and AI alignment sit downstream of existential risk studies, which evaluate extinction risks facing humanity. Unlike the superintelligence risk, many others can be assessed and bound. Take Gamma Ray Bursts (GRBs), for example. These are intense beams of gamma radiation moving at the speed of light. If any were to hit Earth, at minimum, all life on the planet would be threatened, and it’s hypothesized by some scientists that a GRB might have been responsible for at least one previous mass extinction event. One study put the odds of a GRB at about 50% over any 500-million-year period. In 2002 and again in 2013, Nick Bostrom excluded GRBs from existential-risk priority on probability grounds. When scaled down to a century, the risk is probably negligible.

But it’s weird that a risk with a number that can be checked, like GRBs, is ignored while the one that sets the global agenda can’t clearly be measured. Yudkowsky has written screeds about how a truly rational civilization would spend all its time focused solely on AI alignment to solve a risk whose numbers mostly live in individuals’ heads. We know GRBs are real and that one potentially hit the Earth in the deep past, but imagine the absurdity of recommending that we dedicate the entirety of GDP to building ever-bigger telescopes for surveilling the sky, deep-crust bunkers, and lead shielding everywhere. If misaligned superintelligence is a far greater risk than a GRB, it is extremely telling that this cannot be cleanly demonstrated.

The refrain of those who resist the mugging characterization is that GRBs can be measured because there is a reference class with data—GRBs have stellar populations and observed star death frequencies. As they’d suggest, simply because a novel risk lacks data does not imply an absence of risk. While this sounds reasonable, that puts such a risk in the same class as any other unfalsifiable claim. Yes, novel risks are truly hard to measure, and demanding a figure is a high bar. But traditions like fault-tree analysis hold that when a risk is unprecedented and resists decomposition, this should widen uncertainty, not license maximal confidence. We don’t have to believe doom is impossible; however, we can only act on the parts of the problem that are tractable.

Limits on prediction and probability do not make present AI systems harmless, but they change the basis on which AI governance should proceed. We should stop treating speculative maximal scenarios as the overriding framework and focus on demonstrated harms, concrete institutional failures, and risks whose pathways can actually be examined.

If AI capabilities are actually improving, do my criticisms matter?

I’ve made it clear in all of my podcast episodes and my writing that I think AI companies have been irresponsible and are making us less safe. The Hugging Face incident shows both why this technical point matters for governance and why the critique is not a denial of present harm: the hack was real even though the behavior emerged from a composite system. Its costs and deployment conditions also make it a poor basis for treating maximal superintelligence as the central cybersecurity scenario. So why spend so much time attacking capabilitists, especially those who want to pause or stop AI?

First, while I agree harms exist, I think they are bounded both by the current AI architecture and by economics. The Hugging Face hack, for example, saw OpenAI burn months’ worth of compute given its VC funding. These are costs the average bad actor can’t easily absorb. Rather than prematurely base our cybersecurity policy on a single unsupervised training run, we should regulate away the negligence that caused this by instituting liability for AI companies and personal liability for their CEOs. Where it makes sense, we will need cyber policy to help address the vulnerabilities that cheaper open-source models can exploit.

Most importantly, however, I am deeply concerned that uncritically granting capabilitists’ ontology is the far graver harm. I won’t pretend I’m unbiased, as references to my deceased rationalist partner are sprinkled throughout this post. I have been unfortunate enough to witness firsthand what taking these ideas seriously does to vulnerable people, and the rationalist community itself has started to recognize this problem. This is genuinely a dangerous worldview that is consuming very bright and compassionate people, having them spiral into crisis. Beyond that, we honestly have no good reason to believe any of this is true. Not just because it’s implausible, but because these narratives are rooted in an ontology that imagines away complexity. We have no reason to think that, in models at least, intelligence works like a dial on an amp that you can simply crank up to produce the outcome people like Yudkowsky fear/want.

I discussed earlier this year how the effective altruist and rationalist groups (including Yudkowsky’s MIRI), and Anthropic itself, did a flashy media tour with Bernie Sanders. The end result is that Sanders has suggested nationalizing AI labs and has codified Yudkowsky’s interpretation of Superintelligence in a bill. Simultaneously, AI labs are playing red light, green light by offering conditional pauses to stave off the bubble and signal what future policy proposals should look like. I’m very worried that policies built on statements, benchmarks, and assumptions supplied by industry insiders will promote capabilitist ontology and cement the wrong approaches to regulating this technology. By embracing this framing, we are implicitly accepting that AI can only exist in its current form, regulated by the capabilitists actively developing the technology.

While I have nothing but the harshest criticisms for Yudkowskian and other maximalist views, I am more sympathetic to views focused on short-term harms and gradual risks stemming from governance failures. Paul Christiano, who has debated Yudkowsky, is illustrative of the latter. Christiano’s concern, broadly, is gradual disempowerment caused by offloading responsibility to more capable machines. To that end, Christiano has worked on technical solutions, like reinforcement learning from human feedback, while highlighting the role of governance in steering us away from dangerous outcomes. Additionally, I see many of his concerns as echoes of problems in market capitalism. While I don’t fully agree with Christiano’s characterization of AI alignment, the fact that he prioritizes governance and tractable problems makes him more approachable and reasonable than maximalists.

Why does belief in superintelligence persist?

Given the diversity of superintelligence views, it is really hard to conclusively answer why this belief persists. As I said at the beginning of this post, its seeds predate most of our lifetimes, starting with people like von Neumann and IJ Good. While I can’t speak for everyone who holds this belief, within the rationalist community there are certain dispositions and norms that lead people to see the world as inherently easy to measure. If you’ve made it this far you probably noticed this theme throughout the superintelligence argument stack.

Most of the claims I identified are not inherent features of computer systems. Intelligence as a measurable predictive power that can be stacked like lego bricks, then harnessed outside of neurons to turn the world into a chess game, is a worldview. It is a very reductive, naively engineering-minded set of assumptions about how reality should work and was shaped by the similarly engineering-minded 20th-century academics who laid the groundwork for this idea. Until recently, rationalists have viewed solving superintelligence solely as a technical problem, and solutions like my deceased partner’s are meant to mathematically validate that alignment is possible. But even if you believe that we can spawn all-knowing demons from silicon, the processes and resources that enable this are managed by humans. Whether you believe in superintelligence, normal AI, or something in between, governing AI was always going to be a political fight.

Combine this with rationalists’ other norms. Rationalists love assigning expected values to things and even betting on them. They are the origin of prediction markets and arguably the mainstreaming of cryptocurrency. As practices like P(doom) illustrate, they have a compulsion to assign subjective probabilities to events no matter how unlikely. People are often rewarded for contrarianism and for calling out possible black swan events in advance. From what I can tell as an outsider, the only bar to admissibility is that a claim be logically conceivable. Many claims are logically derivable from some set of axioms; however, that doesn’t make them real.

I can also see why superintelligence has appeal outside the rationalist community. We truly are in confusing times, and, oddly enough, beliefs like it provide a perverse form of certainty. Whether you’re trying to build it, align it, or stop it, this is a focal point where you can direct your efforts to shape the future. It is the mother of all problems; solve it and the rest might follow. Personally, I don’t know whether I can believe there’s just one problem to solve; I watched my partner live under duress with that belief for almost a decade. She gave everything to AI alignment work, but sadly I fear that all projects, policies, and activism predicated on this idea are on ontologically shaky ground.

To be fair, I have watched parts of the alignment field move toward a more holistic notion of alignment, focused not on individual models but on subsystems, environments, and even institutions. This is great to see, but this view already existed in fields like AI ethics as well as science and technology studies. This move, along with the migration of alignment out of AI safety orgs and into formal organizations working on scoped, well-defined problems, might yet move the field away from its pre-paradigmatic work. The sooner that happens, the sooner this inane belief propping up the global economy can go away.

TL;DR & summary

• “Superintelligence” is an ontological claim — a statement about what future machines will be, not a measurement of what current machines can do. Its only evidence is extrapolation from shallow but exciting trends, a pattern that has repeated across the history of AI.
• Belief in superintelligence has traveled from mid-century thinkers like von Neumann and IJ Good through the rationalist community into the global economy. It’s now in Anthropic’s IPO prospectus, Sanders’s ban bill, and Trump’s rebrand. But “superintelligence” isn’t one belief; it’s a family of them, sortable by four questions that separate accelerationists from Kurzweil from slow-doom believers:
  • Better than humans?
  • Dangerous?
  • How soon?
  • How much more capable?
• This post audits the maximalist version — Eliezer Yudkowsky’s, as codified in IABIED — arguably the original source for modern superintelligence beliefs. This belief revolves around four core ontological claims:
  • Intelligence as a universal metric that maps to novel capabilities.
  • Cognition as a type of software.
  • Knowledge as cleanly decomposable.
  • AI can develop any goal and will resist attempts to control it to preserve that goal (orthogonality and instrumental convergence).
• Every claim is contestable:
  • No known measure of intelligence works the way proponents need it to, and key benchmarks have ambiguous objects of measurement.
  • The evidence reviewed here does not clearly establish cognitive or goal-directed states in LLMs; claims about those states depend on inferences from external behavior.
  • Actual doom scenarios are implausible. They require a level of accuracy and precision approaching perfect simulation without interaction in the real world.
  • AI doom relies on subjective and unauditable probabilities to feel urgent.
• However, the stack doesn’t even need to be false for the doom case to fail. A misaligned goal still has to be executed. Execution means becoming Laplace’s daemon: an active experimenter in the physical world, not a passive knower.
  • Daemons move at the speed of the world: trials fail, evidence takes time, humans react to pandemics and unauthorized factories. Even granting the whole stack, plus billions of dollars and years of undisturbed experimentation, not everyone dies.
• The science of AI alignment is pre-paradigmatic, relying on philosophical or ontological claims rather than established methods in established fields like computer science. This has greatly influenced the speculative nature of these arguments.
• Belief in this stack is a worldview that reduces intelligence to stackable lego bricks to solve a single problem: becoming smart enough to see the universe like a chess game. But the processes that would enable any of it are managed by humans, and governing AI was always going to be a political fight.

Share this post on social or via email!