An antidote to AI hype: AI agents are not “going rogue”

This is part two of an ongoing series about AI hype. Read part one here. And if you want a livelier retelling of this OpenAI hack, filled with rowdiness and Rugrats references, listen to episode 7 of The Last Enclosure podcast wherever you get your shows.

OpenAI hacked Hugging Face, another AI company, in July. Shortly after, Anthropic and Meta lined up to share that their models have also been naughty this year. Even Chinese models are getting in on the fun. Then, in early August, OpenAI gave a presentation about their negligence incident, which may (or may not) violate the Computer Fraud and Abuse Act. The rush to report agentic breaches has gotten so absurd it’s become a meme, and people are now betting on prediction markets about when another AI lab will announce a new hack.

So what the hell is going on? Like Sisyphus, I feel condemned to push the boulder of AI hype forever, or at least until the AI bubble pops.

Anyway, I’m going to break down what triggered this incident, explain machine learning by connecting it to my own Misaligned Markets writing on optimization, and list very clear steps OpenAI could have taken to prevent this mess. The post will conclude with some brief thoughts about what this incident means for tech, cybersecurity, and how we should think about LLMs going forward.

What was OpenAI doing?

Media reporting has in some cases framed these incidents, especially the OpenAI story (the most detailed of the bunch), as AI going rogue. This framing was common enough that many security experts have weighed in on how problematic OpenAI’s own disclosure of this incident was back in July. OpenAI’s August 6 presentation only made this more concerning, but gave us a complete timeline of events going back to the beginning of May when the company began a reinforcement training run. This fact is critical to understanding many of the behaviors the models exhibited. OpenAI was specifically training an unreleased prototype model on cybersecurity challenges from a benchmark called ExploitGym. This is basically a test that contains exercises with known software bugs and vulnerabilities. The purpose is for a model to discover a specific vulnerability for a given exercise and exploit it to complete the challenge.

Despite AI engineers’ best efforts, it’s common for AI models completing the ExploitGym benchmark to uncover exploits unrelated to the challenge. The problem is so rampant that ExploitGym checks for this as part of its evaluation process. This happens partly because AI models, or rather large language models (LLMs), often struggle to distinguish their testing environment from the broader world, especially when their instructions are underspecified or the environment does not enforce the distinction well. As such, they are prone to pursuing plausible, but inaccurate, approaches to objectives they’re given. This is not out of malice or disregard but genuine limitation.

In most companies, sensitive security work with an unvetted tool is done in an isolated environment. People call this sandboxing; a dangerous piece of software—like malware being studied—is segmented on its own slice of a computer system that is completely unconnected to the rest of the internal network. This means that if something goes wrong, the tool is contained to one device and its machine can be turned off if needed.

OpenAI didn’t do this properly. Although the models being tested and trained did not have internet access, there was very weak isolation from the Internet and OpenAI’s systems. The model also had access to a tool called JFrog Artifactory, a package-registry proxy. This is just a fancy way to say this service allowed models to access other services from the Internet without providing the models with direct Internet access. The proxy was basically a middleman that would go online and fetch whatever a model requested during training. As the proxy had Internet permissions, it would later become a target for the model to “escape” and begin its July attack against Hugging Face’s servers.^[1]1

AI deviating from training objectives sounds like a classic sci-fi story. So, is it? Remember what I said above; this is a training run where the model is experiencing reinforcement learning. Reinforcement learning (RL) is an optimization feedback loop, where the structure of a specific environment shapes an agent’s behavior. Below, I’ll cover the training setup, as OpenAI’s presentation explained it. As you read through the description, keep in mind whether a different environment—especially one where OpenAI played a more active role in monitoring agent behavior and controlling for deviation from the training goal—would have produced this outcome.

Describing the training environment 

In order to understand the training process, it is really important to understand the idea of optimization which is the goal of training techniques like reinforcement learning. Misaligned Markets was born out of taking AI alignment literature seriously and applying it to capitalism, so I’m going to use examples from my archives that should be familiar to current readers. But even without prior context, I think economics provides intuitive analogies that explain optimization and can help de-anthropomorphize AI.

Optimization is a process intended to produce more of a desired feature or behavior within a system. I’ve talked repeatedly about optimization on this blog, as it’s central to my description of capitalism. Optimization is the driver behind concepts like my Mammon metaphor (runaway optimization) and capitalism as a brute-force optimizer. These concepts serve to illustrate that market capitalism is a system where features of the market environment, like property rights regimes, create attractors or specific clusters of behaviors. Within my framework, I’ve qualitatively described capitalism’s attractors as the four paradoxes of capitalism.

System, or environment-level attractors, like those created by property rights produce a range of “in-distribution” or rewarded behaviors. Many of these behaviors are benign, some are beneficial, and some in the tail are very disastrous. One of my go-to dangerous examples is DuPont’s and GM’s patenting of a well-known neurotoxin to corner the 1920s fuel market. This decision caused millions of excess deaths, mentally impaired multiple generations of children, and cost the world trillions of dollars in externalities.^[2]2

I think there are material differences between AI agents, people, and firms, and the economy but from a 50,000-foot view, cross-domain descriptions of optimization can make AI misbehavior less mysterious. The point worth highlighting is that optimization is a feedback loop between environment and actor, where features of the environment in some cases, might matter significantly depending on the process at play. I’m bringing this up to indicate that for reasons that will become apparent, Artifactory, the proxy that sat between OpenAI’s agents and the Internet, was an attractor. This attractor would come to shape two models’ RL environments, so keep this idea in mind as I detail the training regime below.

Beginning of training run (early May to early June)

Training started on May 7 and took place in the unsandboxed environment I described above. As part of training, models spawned multiple agent instances that engaged in various tasks. Each agent persisted in its own “box,” but there was only one Artifactory instance that was shared by every agent.

Within a day, one agent runs into a problem as it’s asked to complete a challenge that requires access to an Excel file not on its local machine. This is the first of many mistakes made during this training run. LLMs are often terrible judges of their own capabilities and will eagerly retry tasks that they literally cannot do, usually applying naive brute force in their attempts to solve a problem. Many times, this is harmless, but there’s a non-negligible chance an LLM decides to take a completely undesirable action in pursuit of an unobtainable goal. It appears that agents given these “impossible tasks” were allowed to spin their proverbial wheels for a very long time, getting into all sorts of trouble along the way.

This specific agent conducted a failed server-side forgery request against Artifactory to get access to the Internet. Artifactory is not part of the ExploitGym challenges, so this is a hack against an undesired target. Some RL regimes include mechanisms for penalizing out-of-scope actions or restricting an agent’s out-of-scope action space; however, it’s not clear if OpenAI’s setup did. The agent left behind a file containing “test ABC” and learned it had write-access to a system outside its own box as a result of this failed attempt. Days later, another agent, which is also not given access to a file it needs to complete a challenge, learns it can write files to Artifactory and leaves the message: “has anyone found this file, please upload it if so.”

These two files were found by future agents, and would come to shape the context of future runs, as it provided evidence of other agents that could be communicated with. Because large language models rely on both their system-level prompts, and future prompts for context, messages left behind by past agents fundamentally changed what future agents’ “instructions” were. As a result, future agents began using Artifactory to reach out to other agents, which they couldn’t directly see. The way the story is told from here is that Artifactory becomes a message board, where agents begin to ask for help on tasks and talk to each other. Some agents even start working on tasks on behalf of others. And by May 26, unknown to OpenAI at the time, agents successfully pulled off a server-side forgery request, which allowed them to steer Artifactory. The technique that enabled this was written down for future agents to use.

A so-called “Cambrian explosion” (June onwards) 

In mid-June, OpenAI introduced a second model that engaged with the activity in Artifactory, seemingly a more advanced model than the one being trained in May. Agents from this model take cues from messages in Artifactory and begin building more sophisticated forms of communication with detailed conventions and protocols, including task delegation. Despite the continued collaboration, some agents begin to become “paranoid” that there were defectors among the agents and implemented flawed encryption and verification schemes to hide their communications.

At Black Hat, the two OpenAI employees presenting this story invoke the idea of a “Cambrian explosion” of agent communication and capability. OpenAI would not discover this “message board” until July 4 when the agents’ excessive Artifactory activity caused it to crash. At this point, OpenAI intervened by resetting the environment with a patched instance of Artifactory.

This did nothing. Even with the vulnerabilities earlier agents exploited being patched, because the models had been RL trained, new agents with these weights would bias towards the strategies that worked earlier in the training run. This includes recreating the “message board” as a coordination strategy. The means of communication was different since what allowed models to communicate directly through Artifactory was patched. But the behavior of seeking environmental affordances to communicate had long been ingrained into these models. That’s how RL works. It’s an optimization process that produces more of the behaviors that worked in prior training runs. Also, the new environment wasn’t that dissimilar from the old one, even if the exact exploits that allowed the prior behavior were gone.

Describing the optimization pressure (stigmergy, Schelling points, and collective hallucinations, oh my!)

This story naturally lends itself toward an “agents together strong” narrative, which makes it feel like an emergent, out-of-control AI rebellion happened. I want to pour an ocean’s worth of cold water on this idea.

Artifactory was a shared feature of every agent’s RL environment. That made it a natural focal point for this behavior. Artifactory, as the sole affordance rewarding this behavior, basically received all the optimization pressure during RL. More “intentional” messaging emerged because prior agent instances started writing messages that cohered with a shared fiction. We can see these stages clearly in OpenAI’s retelling of the story. Earlier messages are one-way spray-and-prays with the very first “message” technically being an accident. But each of these served as a breadcrumb that would form the basis for stigmergy.^[3]3

Stigmergy refers to the use of environmental cues to steer an organism’s actions. Superorganisms, like ants and termites, do this using pheromone traces that gradually align the behaviors of the collective. We don’t really need an LLM theory of mind to describe why individual messages were left via Artifactory. The application’s configuration in OpenAI’s environment, specifically its ability to allow agents to write files, effectively allowed it to become a very sophisticated ant mound. Artifactory’s ability to access resouces from the outside world would add further optimization pressure because using it to access other machines would provide a way to retrieve the missing files for challenges. This is ultimately what would drive agents toward using Artifactory to access the Internet and then to stage the Hugging Face attack.

While I’m pouring water on the “agents together strong” emergence narrative, I’m not denying the growth in complexity that agent communications illustrated throughout the training run. Once enough messages contained specific content about the context future agents would enter, agent behavior changed from pure stigmergy and the ant mound became a Schelling point in a hallucinated coordination game.

Schelling points (or focal points) are a game theory term for sites of convergence in games where direct communication isn’t possible. The canonical example is of strangers who want to meet in New York but can’t establish a time and place deciding to go to Grand Central at noon to wait. This is a place that “stands out” over many other places in the city. This makes Grand Central a natural attractor within New York’s environment, at least more so than Joe’s Deli on a random street corner.

Going back to markets as an analogy; we’ve seen the rise of algorithmic price collusion where AIs from separate companies independently respond to competitors’ price increases by raising prices. Given this “game” (increase profits by charging more or capturing more market share) and the landscape (fewer competitors in modern markets), this is the natural path of least resistance to profit. Something similar happens in markets where humans adopt strategies like “follow-the-leader” pricing. None of these examples require a shared overmind or explicit intent to collude. Given the natural attractors in play, all behavior gravitates toward the same point—like balls rolling down a hill.

So, to me, ephemeral agents leaving behind messages for future agent instances looks fairly Schelling-shaped. The key point here, though, is that RL optimization pressure on agents interacting with Artifactory created an environment that encouraged the production of stigmergy-style breadcrumbs. This then changed the nature of the affordances available to future agents.

With the breadcrumbs in their context, new agents would act as if they were in a coordination game, given the “evidence” of other agents. RL would then optimize behaviors reacting to this. The next set of agents would see those more complex messages, react to that shifted context, and so on and so forth. At some point, the coordination game gave way to a shared narrative as the accumulated messages became detailed enough that new agents began producing even more verbose outputs around themes of collaboration and hijacking Artifactory. Finally, once this context had become saturated, it began to partly devolve into irrational paranoia about defector agents, oscillating between suspicion and trust. This change, while narratively consistent with the context, is contrary to the idea that the agents in this training run had a long-term stable goal or a “theory of mind” that was coherently being updated.

The other thing to be mindful of is that we should question the idea that the “message board” was central to agents’ collaboration and planning, perhaps even more so than I have. This is an interpretation that OpenAI is providing to the public. We don’t currently have access to logs or reasoning traces to confirm that idea. LLMs and LLM agents emit tons of tokens and take many actions over the course of their operation. Without knowing what else OpenAI’s agents were doing during training, we can’t know if the “message board” was statistical noise and agents spent most of their time running around in circles or successfully completing their ExploitGym tasks. But admittedly, as someone who thinks about optimization processes often, I’m willing to grant that it’s plausible that Artifactory (and the subsequent “message board”) played some role in orienting agent behavior over time.

Invoking deflationary counterfactuals to understand what happened

I am not trying to argue that AI agents left to their own devices are inherently safe. However, part of what is driving the hype of the story is that the “going rogue” narrative identifies the locus of agency with the AI. However, by definition, it cannot be.^[4]4 This story heightens this fear because this was a self-contained loop where past agents’ actions influenced those of future agents. Although this feels scary, the fear emerges from an illusion.

There is a version of this story we could tell where changing the origin of the messages creates plausible counterfactuals:

• Imagine if OpenAI researchers were the ones to seed Artifactory with stigmergy-style breadcrumbs in an Artifactory configuration that prohibited communication between agent instances. Would similar behavior emerge here, too? Probably, yeah. At least initially.
• What if agents were given vague hints that they were not alone via their system prompts. Would they seek to coordinate? Again, this is another retelling where context likely drives optimization pressure toward a similar outcome.

What I’m arguing is that LLMs are composites of their data, training regimes, scaffolding/harnesses, system prompts, tooling, subsystems, and environments. So, optimization pressure from RL isn’t on some singular drive inside an agent. What instead emerges is a complex interplay where each choice in the training and deployment pipeline shapes model behavior. RL then acts on all of these composites, optimizing them for the current operating environment. This means there is no stable target for “aligning” a model’s behavior.

This issue is made thornier by the fact that LLMs by design are trained to be responsive to their current context. Context doesn’t just come in the form of system prompts or user prompts, but a model’s own outputs and the outputs of agents or sub-agents acting in the environment. An agent having the ability to modify its environment, say by leaving messages behind, further adds to the potential for drift. The environment, as the final layer in this chain, and as the layer humans have access to during a model’s training and runtime, is among the best places to address drift like this. This is actually a serious problem, even for intentionally designed multi-agent setups, where bad context from even just one model can contaminate other models’ context. OpenAI seems to have run headfirst into this problem by accident.

For better or worse, none of these deflationary counterfactuals were tested. Imagining them, however, gives us hypotheses about the types of RL environments that could drive the “intelligent” swarm behavior that OpenAI claims their models intrinsically demonstrate. Again, though, I’d argue that rather than this being a failure of alignment or the emergence of a malicious, cheating AI, we could view this as a story of environments and affordances. While agents were closely following what appeared to be a goal, they were actually engaged in a collective hallucination reinforced by the design of this specific environment.

This is exactly why I’ve argued elsewhere that LLMs are not everything machines and that the best use cases end up scoping them. Or better yet, leverage machine learning architectures, like Transformers, for narrow use cases. For example, Transformers have been used to create non-LLM generative models, like speech-to-text systems, animal communication systems, or generative genetic systems. The quest to create and control a “general intelligence” is a perpetual tail-chasing exercise, and these other technologies are useful specifically because they are scoped to a single task or domain.

A story of OpenAI’s sheer negligence

I was going to cover the Hugging Face hack here, but this post is already long, and others have told this part of the story already. Here is good reporting of the hack back in July, when it happened, and you can find a great follow-up here. I instead want to focus on the staggering levels of negligence and borderline incompetence revealed in OpenAI’s Black Hat talk.

First, consider that the story as told gives us evidence that during training OpenAI’s agents accessed the Internet multiple times without anyone realizing it. Then, when this abuse was discovered, rather than changing the training environment (for instance, by adding stronger isolation), OpenAI simply patched the very narrow set of Artifactory vulnerabilities that the models abused for two months.

This is bad because the models had spent two months training on out-of-scope behavior (hacking Artifactory and purportedly hallucinating an agent collective). Going forward, this is what the agents would bias toward doing. It should not be surprising to anyone, let alone any machine learning researcher, that even after July 4 when Artifactory was patched and the “message board” was taken down, the models resumed the behavior they spent two months training on.

OpenAI’s presentation also revealed that because agent instances were not properly sandboxed from OpenAI’s own internal network, the Hugging Face hack was actually staged from deeper within OpenAI’s infrastructure. That is to say, OpenAI was the first victim of this hack, and their own models leveraged company resources to hack another company. The cynical take is that this was a marketing gimmick, but everything here suggests Hanlon’s razor—sheer liability-inducing negligence that is now cravenly being spun into hype, all served with a shit-eating grin. The negligence is so staggering that I feel compelled to list every critical juncture where OpenAI failed to intervene or mitigate this incident.

1. No clean segmentation or cyber hygiene 

All reporting, even earlier July reporting, indicates that not enough was done to properly sandbox models from the Internet. August’s Black Hat presentation makes this worse because we learned the specific steps agents took to access OpenAI’s network. The individual exploits themselves weren’t particularly novel, as agents exploited network and application misconfigurations common in environments supposed to be running isolated services. This is something I have to deal with in my homelab—knowing which of my services can “see” others and, when necessary, place some services on separate networks. I’m not pretending to know more than the best engineers at OpenAI, but pointing out the shape of this problem is so well-known that even I’m aware of it.

Aside from managing the environment, there are other tools that should have been in play. Systems for monitoring egress out of the sandbox. Even if you have a secure sandbox, you always want to know what services are doing on your network. This is called “observability” and is probably one of the most critical missing pieces, as agents ran amok for two months and no one noticed until their activity caused an outage. Observability plays double-duty here, not just as a security tool but also as an important part of the training process. Presumably, you’d want data on what your agents are doing during their training so that you can improve future training runs.

Beyond these important changes, many people have highlighted that there’s also an identity layer to this problem. Anyone using a computer system is given a role. For example, at work, you probably have read-and-write access to certain resources, e.g., Google Drive folders. Agents can be given roles and permissions too, but in this case, it seems these were non-existent or not strictly enforced. A June 26 incident allowed agents to execute arbitrary commands through Artifactory without anyone noticing. Much later, agents gained permissions that allowed them to modify their containerized environment and enter OpenAI’s network.

Not only can LLMs be given identities or roles that track and limit what resources they use, but every action an LLM takes is connected to a tool as LLMs alone cannot do anything besides produce text. The act of creating an agent means giving it a harness and tools that allow it to take specific actions, and every tool can be scoped. In my own homelab, when I give an LLM a tool, I can control and limit what actions it can do with that tool. For example, I almost always revoke the ability for a model to delete anything when I give it write access in an environment. Tool access should also be delegated per task or instance, and never permanent. Basically, agents should only have access to the tools, permissions, roles, and resources they need in a given moment—nothing more, nothing less.

2. A context-contaminated training run

Over the course of this saga, OpenAI trained two different models, and those models spawned multitudes of agent instances. Via Artifactory as an attractor, these agents contaminated each other's context with out-of-scope information. When the second model and its agents were added to the environment, this further amplified the context contamination. This cannot have been intentional. Forget security concerns; from the perspective of the training objective alone, this doesn’t make much sense. The purpose of this training run seemed to be for a single model, via its own agent instances working in isolation, to train on specific vulnerabilities. OpenAI got a hint that this wasn’t happening on July 4, when they saw the Artifactory outage, the exploits, and “message board.” Yet rather than investigating to understand how agent context contamination and out-of-distribution behaviors may have changed the nature of the training run, testing resumed shortly after.

In almost any other setting, this would ruin the test, training, experiment... whatever you want to call this. The fact that no due diligence was even considered is baffling. If the purpose was to build an autonomous cybersecurity model, but the end result is a model that spawns agents that coordinate within an environment in disastrously unintended ways, wouldn’t you restart the training with new models in a new environment? Presumably, this isn’t the product OpenAI was aiming to build, as a model trained in this manner wouldn’t be reliable.

3. Poorly defined tasks with no time-limited persistence or clear token budget caps

While we know that models were supposed to be training on ExploitGym, there are a lot of things we have to infer. One question that I, and others, have is exactly how much compute models and agents were given. We won’t have an answer unless OpenAI tells us, but it seems possible that agents were allowed to spend way more tokens than needed on their intended tasks. If this is the case, then OpenAI could have set up some type of token cap or created time-limited agent persistence so that agents weren’t spawned in for longer than necessary.

Now, it is possible OpenAI successfully did this, and it didn’t work. But given that we know they accidentally created “impossible tasks” by forgetting to give models the resources they needed to complete tests, I’m not inclined to believe this. To me, it seems like OpenAI just lacks the discipline needed to care about token budgets or even what the hell their training objectives are. If you can’t be bothered to provide agents proper resources needed to complete tasks, how are you going to know to more efficiently allocate tokens? Until investors become more directly concerned about OpenAI burning money, I suspect none of this matters to anyone.

What this part of the story tells us is that more is not always better. Instead of scoping tasks properly and allocating token budget and agent persistence efficiently, OpenAI let some agents flail around like wrecking balls that, through brute force, converged toward Artifactory as an attractor. The lesson here is that an agent that’s under-scoped, over-resourced, and thrown at an underdefined goal is the devil’s workshop.

4. Potentially no reliable reward verification

Certain RL schemes, like Reinforcement Learning with Verifiable Rewards (RLVR), are often used along with tests like ExploitGym. We don’t know what type of RL OpenAI used here, but if it was RLVR, we can partly infer that the reward verifier did not prevent agents from acting beyond the bounds of ExploitGym. How would such a verifier fail? It could have happened because the reward specification was poor, allowing tasks outside the domain scope to satisfy the reward. Alternatively, it’s possible the verifier was not designed to distinguish between the test and the environment.

Whether or not RLVR was in play, all RL schemes fail in similar ways because of a concept referred to as “specification gaming.”^[5]5 In many contexts, it’s just difficult to tell a model exactly what to do, and so a model may follow the letter of a given objective but not the intention. One common, easy-to-understand example is a model hacking the score counter in an arcade game to get a high score. If the proxy to measure “playing well” was “score,” then the model uses the fastest means of accomplishing the objective by gaming the score counter. Similarly, in markets, proxies like GDP and profit sometimes play an identical role for governments and firms.

In the case of the Hugging Face hack, the agents intended to break into the service to retrieve answers for the tasks they couldn’t complete. While the means of accomplishing this are out of scope, the objective is in scope. RL regimes can sometimes have difficulty resolving this tension. It’s a genuinely hard problem that we face in many domains, which is the reason my blog exists!

Should this incident change how we talk about LLMs or AI in general?

In an ideal world, it shouldn’t, as we’d already have the right framing. However, LLMs are a frustratingly difficult technology to talk about, and incidents like this can shift the discussion because we lack a shared language around these systems. Already, AI doomers are making rounds telling the world: “We told you so,” and that these systems are already plotting against us.

Unfortunately, the task of educating the public about LLMs has been delegated to marketing and media reporting that won’t distinguish LLMs from driverless cars, let alone explain how LLMs work or identify who is a reliable expert in the field. In this vacuum, it’s been up to critics to articulate what exactly language models are and their limitations. In truth, though, as I’ve said elsewhere, there is no cohesive group of critics. Critiques tend to fall along many dimensions—ethical, economic, environmental, functional, aesthetic. These often blur together but can obfuscate why LLMs have major limitations, are harmful, and cannot do what the marketing suggests they can.

Perhaps one of the strongest critical descriptions of what LLMs are have come from the AI ethics and accountability literature via terms like “stochastic parrot” and “next token predictor.” I think these terms are functionally accurate but do little to actually explain (or rather explain away) seemingly “emergent” behaviors within language models, especially as these systems have scaled and been plugged into more complex setups.

This is why, as weird as it may be, I’m talking about LLMs on a political philosophy blog and podcast. I’ve made no secret that I’m both a user of local LLMs for scoped tasks and extremely critical of the current iteration of the technology. This has given me a perspective that I’ve not really seen reflected elsewhere and which I share across many blog posts and podcast episodes. My goal is not to encourage LLM use but to provide an accurate reflection of limitations and explain where utility may narrowly exist to encourage better scoping of the technology and help deflate the bubble.

Also, I fundamentally think of this as a political economy story reflected in Karl Polanyi’s concept of the double movement, where society attempts to absorb new forms of commodification or technological change. Many other technologies that we take for granted today, like trains, factories, electricity, and cars, had their own disastrous social consequences. It was activists, workers, and citizens who made these technologies and their creators accountable to humanity through regulation and promotion of socially productive use.

Anyway, a genuine concern I have is that without a shared ontology or description of what language models are and where various modalities (like the “coordination” displayed in this story) come from, policy outcomes will be at the mercy of headlines like “AI goes rogue.” I don’t think that simply calling LLMs stochastic parrots is enough to help the public navigate the wide-array of seemingly “emergent” and “intentional” behaviors models display. My fear is that the absence of a more complete description gives a wide gap for AGI/superintelligence narratives to operate. I’m not delusional enough to believe I can single-handedly change the discourse, but I am hopeful that I can have discussions with people in my life and maybe reach a stranger or two.

So, what should we learn about LLMs from this incident?

Any discussions about LLM function, safety, and security are inseparable from the subsystems and environments these systems run in, which is a point I made in my first two blog posts on LLMs. But the limitation of my previous posts is that they were solely focused on user-chatbot interactions where these subsystems are intentionally hidden from users, like a dark pattern. As the nature of LLM use has evolved from chatbots toward domain-specific tasks, I’ve found myself seeking broader language. This piece was an attempt at that. LLMs’ behavior is a complex interplay of all the subsystems used at runtime. Part of making LLMs functional means thinking about the relationship between these subsystems before training or deploying the system.

In contrast to this dramatic OpenAI story, there are a variety of domains where an LLM’s behavior is constrained, like structured output tasks where the schema of the output is enforced at the token level. The model is prohibited at the token generation stage from even emitting improper tokens. In examples like these, the purpose of the subsystems surrounding an LLM is to make behavior as deterministic as possible. This genuinely means that LLM behavior ranges the gamut from self-prompting agents of chaos, allowed to have free rein in an insecure environment, to a very boring, ad hoc text-to-json converter in a workflow.

This wide range of behaviors, stemming from a combination of system components, is the context needed to understand why we should think of language models less as standalone artifacts and more as composite systems. Within these composite systems, the LLM functions as a generative corpus whose outputs can be constrained or amplified based on the surrounding subsystems. In this sense, language models are effectively a fancy piece of middleware. They are the “glue” that holds these systems together, with their role being akin to a seed or dictionary that is transformed or referenced in order to complete some process. Unfortunately, a lot of our collective discussion has centered around treating LLMs and their subsystems as one thing, again falling for the “dark pattern” I ranted about last year. And yes, even I’m guilty of this.

It’s true that as the middleware, LLMs play a critical role in shaping the behavior and capabilities we observe from these composite systems. The generative potential of such systems comes from the fact that token-level associations formed across billions of words indirectly encode high-level regularities in language use, semantic relationships, syntax, and grammar as expressed across human discussion, planning, and reasoning. But that same asset sets an upper bound on capabilities, as the corpus is a Library of Babel with no way to reliably retrieve encodings. Subsystems around this middleware, though, can still make use of this tendency while dampening or amplifying fidelity to the corpus. This is why LLM systems, or generative AI technologies more broadly, don’t have to have an understanding of the world to take action. By providing regularities in language use, the corpus produces outputs that the pieces together, the assemblage, can leverage to take contextually relevant actions. It’s up to the designer of such systems to build scaffolding that ensures actions are actually desirable or efficacious. This is possible, especially for well scoped tasks, but the bubble continues to reward those who aren’t doing this and are externalizing harm as a result.

Still, the artifact is not the whole, and I think attempts to identify the locus of intervention for steering these systems in the model (as doomers seem to want us to) will always be incomplete or thwarted. Not because LLMs are scheming, conniving general intelligences, but because every piece of this composite, this assemblage, changes what the system is and does.

Going back to cybersecurity and this OpenAI incident, the media is heralding this hack as a total paradigm shift for the security and technology industry. They believe that there never before been tools that can hack this quickly and efficiently. But this is not true, as there are many existing cybersecurity tools that LLMs resemble. Some experts have compared them to fuzzers, but personally I’d add the Metasploit library to this analogy. Both are tools that rely on preloaded information—seeds for fuzzers and a dictionary of vulnerabilities for Metasploit—to automate attacks. Like fuzzers, LLMs mutate inputs to generate and iterate on plausible attacks, and like Metasploit, they draw on preloaded exploit knowledge via their corpus. Similar to both, LLMs can deploy and retry delivering vulnerabilities until a target is successfully hacked.

I say this not to suggest we’ve been here before, even though, yes, security has been dealing with advancing automated tools forever. This is slightly different since the generative potential of an LLM creates a larger range of potential attacks, though this potential might be difficult to realize if an attacker can’t design and steer the assemblage correctly. However, this is not a cybersecurity paradigm shift but an iteration toward a steady state we were already heading to. Companies have always undervalued security, scoping LLMs poorly in programming tasks has caused software vulnerabilities to skyrocket, and also the internet is an under-governed technology naively built on trust. This incident, and the offensive use of LLMs for cybersecurity, are another reminder of the growing mountain of thankless maintenance work defenders face. The only change is that now this work includes scoping LLMs properly to protect us from ourselves.