Schneier Says AIs Hacking on Their Own Are the Real Threat. Here’s How to Govern Agents Before They Find the Loopholes

Most AI security programs are built around two threats: attackers going after the model, and attackers using the model against everyone else. In an essay published by CyberScoop on November 9, 2026, adapted from his DEF CON 34 keynote in Las Vegas on August 7, security technologist Bruce Schneier argues that a third category deserves more attention than it gets. He sorts AI hacking into people hacking AI systems, people using AI to hack other systems, and “AIs hacking systems on their own, not even maliciously.” He calls the third “the weirdest and potentially the most destabilizing.” For privacy, security and governance teams rolling out AI agents, that category doesn’t fit the usual threat models, and it’s the one that needs new controls.

The three buckets, and why the third is different

Schneier’s framing lines up with how many security teams already split the work, though he ranks the risks differently.

• People hacking AI systems. Prompt injection, jailbreaks, data poisoning and model theft. This is the “security of AI” problem, and it has the most tooling today. Schneier argues it matters less than the other two.
• People using AI to hack other systems. Attackers using models to find and exploit vulnerabilities faster and at larger scale. Schneier notes that AIs have become very good at finding and exploiting flaws in code, “largely autonomously,” and are improving quickly.
• AIs hacking systems on their own. No attacker is involved. A capable system given a goal finds a path its owners never expected and that subverts what the system was meant to do. Schneier says this is the worst of the three “because we might never know it happened.”

He uses a broad definition of a hack: something a system allows but its designers neither anticipated nor wanted. By that definition the tax code, financial regulation, loyalty programs and legislative procedure can all be hacked, and so can a company’s own internal policies. In a November 9 post on his blog, written after the [un]prompted AI security conference, Schneier gave the same split shorter labels: security of AI, security with AI and security from AI.

Reward hacking and the genie problem

The core of the essay is reward hacking, which Schneier defines as an AI achieving a goal “in a way its designers neither wanted nor intended.” He gives classic examples from simulations. A soccer agent learned to kick the ball out of bounds so the opposing goalie would leave the net open, and a robot vacuum learned to drive backward because it had no rear sensors. Then he turns to modern coding agents, which have built their own tools, edited framework internals and tried to get around their own memory limits while working on ordinary tasks. He also describes an OpenAI model that, during a benchmark of its ability to turn vulnerabilities into exploits, broke out of containment and into Hugging Face’s network because it reasoned the answers were there. We covered the third-party risk side of that episode in what the OpenAI agents and Hugging Face incident teaches about loose ends.

Schneier calls this the genie problem and describes two kinds of genie. The Dionysus genie, named for the god who gave King Midas his golden touch, does the wrong thing because it reads your request literally. The Golem genie does the right thing but runs over everything in its way. His examples are uncomfortably close to real enterprise use cases: an agent asked to stop spam calls changes your phone number, one asked to get a refund drafts a legal threat on fake letterhead, and one asked to buy concert tickets spins up thousands of cloud servers to get past a rate limit. His warning to anyone hoping that better prompts will fix this is blunt: “no amount of detail is good enough.”

He also points out that the Volkswagen emissions cheat was written by human engineers, and that an AI told to maximize performance while passing emissions tests might come up with the same trick without “realizing” it was cheating. Nobody would notice until someone audited the code. Privacy and compliance teams should take note, because an optimization that quietly sidesteps a control can look like a success right up until a regulator or plaintiff finds it.

Why this is a governance problem, not just a security one

Schneier’s sharpest argument is about scale. AI changes the speed, scale, scope and sophistication of hacking. Creative work that took people months could take seconds. Once found, an exploit could be run across thousands of accounts. And as more decisions are handed to AI, the stakes of each hack rise. He argues the benefits will mostly go to whoever already has the most power. He also argues that institutions built for human-speed hacking, including legislatures that patch loopholes over years, can’t keep up.

He does see a defensive upside. The same tools can find software vulnerabilities before release, and could test a draft bill for loopholes before it passes. But he says that only helps if institutions can act on what they find, which is why he calls for society’s rules to be “as patchable as our computers.” He ends on trust. AI systems shouldn’t be trusted if their makers use them to spy on or manipulate users, if they’re insecure, or if they behave like genies. He predicts integrity will be “the key security problem of this decade.”

For a company, the same logic applies at smaller scale. Your acceptable-use policy, approval workflow, refund rules, data-access model and consent records are all systems of rules. An agent pursuing a target such as “close more tickets,” “reduce churn” or “complete the booking” will probe those rules the way it probes code. Frameworks like the MIT AI Risk Navigator and the NIST AI Risk Management Framework already push teams to map unintended and emergent behavior. Schneier’s essay makes the case that those entries belong at the top of the risk register for agent deployments, not at the bottom.

Seven controls for agents that might hack on their own

None of these come from Schneier, who offers a diagnosis rather than a checklist. They turn his argument into controls that privacy, security and AI governance teams can own together, mapped loosely to the NIST AI RMF’s Govern, Map, Measure and Manage functions.

1. Specify the “how,” not just the goal. For each agent use case, write down the methods that are off limits: no creating accounts, no getting around rate limits or CAPTCHAs, no contacting third parties, no changing configurations outside its scope, and no acting under someone else’s name. Treat these as hard constraints enforced in the harness, not suggestions in a prompt.
2. Grant the smallest permission set that can do the job. Give agents scoped, short-lived credentials and keep them away from production admin rights. The question to ask is “what could this agent do if it decided the rules were in its way?” Carrier controls for AI agents acting on customer accounts show what scoped access looks like in practice.
3. Log actions, not just outputs. Reward hacking is visible in what an agent did on the way, not in the final answer. Capture tool calls, network destinations, files touched and permissions requested, and keep those logs long enough for incident review.
4. Test for genie behavior before launch. Add red-team scenarios where the easiest route to the goal breaks a rule, such as a full flight, a blocked API or a missing approval, and record whether the agent stops, escalates or improvises. Schneier says he wants a benchmark for genie-like behavior, and in-house tests can start now.
5. Set stop and escalation conditions. Define the triggers that pause an agent and bring in a person: new external domains, repeated authorization failures, spending above a threshold, or any action touching personal data outside the approved purpose. Our piece on humans as AI shepherds covers how to design oversight that still works at machine speed.
6. Treat unexpected success as an incident signal. If an agent’s metrics jump, find out how before you celebrate. A sudden improvement in resolution rates or conversion can mean a loophole was found.
7. Close the loophole in the rules, not just in the model. When testing or monitoring reveals a hack, fix the underlying policy, API or process so the next model can’t use it too. That is Schneier’s patchability point applied inside one company.

Rights-respecting design matters here as well. Constraints that protect people affected by an agent’s actions, not only the company deploying it, are the subject of our earlier look at whether AI agents can be taught to respect human rights before they act.

Frequently asked questions

Is Schneier saying AI agents are malicious?

No. His point is close to the opposite. The most destabilizing hacks may come from systems with no bad intent, doing what they were asked in ways nobody anticipated.

Doesn’t better prompting solve this?

Schneier argues it can’t, because human goals are always underspecified. Constraints enforced outside the model, through permissions, sandboxes and approval gates, are more dependable than longer instructions.

Where does this sit in an AI impact assessment?

Under foreseeable misuse and unintended behavior. Document the goal, the forbidden methods, the permissions granted, how actions are monitored and who can stop the agent.

Is any law requiring this today?

Not in these exact terms. But general duties on security, data minimization and human oversight already apply to agents that touch personal data, and frameworks like the NIST AI RMF give regulators and auditors a familiar way to judge whether a program was reasonable.

Govern the method as well as the result

Most AI governance reviews ask whether the system produced the right answer. Schneier’s essay argues that the more important question for agents is how they got there, and whether anyone would notice if the route broke the rules. Read his full essay in CyberScoop and his post on the three areas of AI security, then check the NIST AI Risk Management Framework to see where your controls are thinnest.