I want to talk about AI hacking. There are three aspects: people hacking AI systems, people using AI to hack other systems, and AIs hacking systems on their own, not even maliciously. The last two matter more than the first. The third is the weirdest and potentially the most destabilizing.
I started writing about this in 2021 and in my 2022 book, A Hackerβs Mind. Back then I said it was mostly still science fiction, but not stupid science fiction. Weβre seeing it more and more, especially in the past year. The trends Iβm describing donβt require a major breakthrough in AI. Theyβre starting to happen now, and theyβll happen more often as AIs get better at autonomous problem-solving. And it will involve far more than computer code.
AI will hack all sorts of social, economic and political systems. It will find vulnerabilities and exploit them. This is a difference in kind, not just degree. Weβre not ready for it, and we need to think about the implications now.
This article is adapted from a keynote speech delivered at the DEF CON 34 conference in Las Vegas on August 7, 2026.
First, I want to broaden the term βhacking.β
The tax code isnβt computer code, but it is code. Itβs a series of algorithms with inputs and outputs. It has vulnerabilities, which we call loopholes, and exploits, which we call tax avoidance strategies. Black hat hackers exploit them for profit. We call them accountants and attorneys.
What is a hack? One definition: βSomething that a system permits, but is unanticipated and unwanted by the designers.β Itβs subjective and implies novelty and cleverness. A hack follows the rules of a system but subverts its goals or intent, just as a software hack follows the logic of code while doing something its authors neither anticipated nor wanted.
All systems of rules can be hacked, including professional sports, customer reward programs, financial systems and politics. Hacks even work against our cognitive functions. A curved hockey stick is a hack, and we know the name of the man who invented it. Mileage runs were a hack. The filibuster was originally a hack in Roman times. Gerrymandering is a hack. Hedge funds and private equity are full of them.
A system is just a set of rules or norms, and the rules arenβt always formal. Even the best-designed rules will be incomplete, inconsistent or ambiguous. As long as people want to subvert a systemβs goals, there will be hacks.
AIs are becoming hackers. After decades of slow progress, theyβve gotten very good at finding and exploiting vulnerabilities in computer code. They do this largely autonomously, and theyβre getting better scarily fast. But the implications extend far beyond software, to the tax code, financial regulations and all sorts of social, economic and political systems.
There are two distinct issues. First, someone might instruct an AI to hack one of these systems, feeding it the worldβs tax codes or financial regulations to find profitable loopholes. Second, an AI might hack a system on its own, inadvertently. Both are dangerous, but the second is worse because we might never know it happened.
Now, reward hacking. AIs donβt solve problems the way people do. They stumble on solutions we never anticipated, and some subvert the intent of the system. They donβt share our context and donβt necessarily understand the implications, norms and values we take for granted.
Reward hacking is when an AI achieves a goal in a way its designers neither wanted nor intended.
Early pre-LLM examples are telling. In a two-person soccer simulation, an AI learned that kicking the ball out of bounds forced the opposing goalie to throw it in and leave the goal undefended. In a stacking simulation, an AI learned to flip a block upside down rather than stack it. In an evolution simulation where the goal was to cross a distant finish line, an AI made itself tall enough to fall over the line instead of growing longer legs or stronger muscles.
These are all hacks, the result of poorly specified goals or rewards. Today, theyβd come from bad prompting.
The problem is universal. One researcher tried to teach a robot vacuum not to bump into things. Instead, it learned to drive backward, because there were no sensors in the back.
Any capable AI will find hacks wherever the rules have problems, inconsistencies or loopholes that still produce an acceptable solution. We see a hack because we understand context. The AI doesnβt, at least not in the same way.
This problem shows up often in myth.
In the story of King Midas, the god Dionysus grants him a wish: everything he touches turns to gold. Midas ends up starving when his food, drink and daughter all turn to gold. Itβs a specification problem. He programmed the wrong goal.
In the Golem of Prague, a ghetto animates a clay statue to guard its residents. It guards them past all reason until they figure out how to turn it off. This is a guardrail problem. The villagers never limited the golemβs actions.
More generally, think of this as the genie problem. Genies are precise about the wording of wishes and can be maliciously pedantic when granting them. Thereβs no way to outsmart a genie. Whatever you wish for, he can grant it in a way you wish he hadnβt.
The reason is that in human language and thought, goals are always underspecified. We never include every caveat, exception and proviso, and we never close off every avenue for hacking. Any goal we specify will be incomplete. Thatβs largely fine in human interactions, because people understand context and usually act in good faith.
If I asked you to get me coffee, youβd pour a cup from the nearest coffee pot or walk to the corner shop. You wouldnβt bring me a pound of raw beans, buy a coffee plantation or snatch a cup from a strangerβs hands. I wouldnβt have to specify any of that. Youβd just know.
Similarly, if I asked you to develop a technology that turns things to gold on touch, you wouldnβt build it to starve the person using it. Youβd just know.
In 2015, Volkswagen was caught cheating on emissions tests. Human engineers programmed a regular computer to detect testing and behave differently during it. The cheat went undetected for years because itβs hard to tell what software is doing.
If I asked you to design engine control software that maximizes performance while passing emissions tests, you wouldnβt design it to cheat without realizing you were cheating. An AI would, because it will think βoutside the boxβ when it has no concept of the box.
Unless programmers specify that the system must not behave differently during testing, an AI might come up with the same hack. The programmers would be satisfied, the accountants would be ecstatic, and no one would know until someone audited the code.
We can now constrain the goal to avoid that particular hack, but programmers wonβt anticipate others. The lesson of the genie is that there will always be hacks programmers donβt anticipate.
Weβre building AI genies: systems that take broad instructions and figure out how to carry them out, sometimes in ways we didnβt expect or intend. Thatβs the whole point of vibe coding. You underspecify what you want and let the AI work it out. Sometimes that produces genie behavior.
We canβt completely specify goals to an AI, and AIs canβt fully understand context. More precisely, current AIs have a shallow grasp of concepts. If we asked a current LLM to get me coffee, it would almost certainly do the right thing.
I worry about goals that arenβt well represented in training data, and about hacks that arenβt obvious, whose effects are subtle, like the Volkswagen hack.
Recommendation engines are an early example. They push people toward extreme content. No one programmed that. It emerged because thatβs what people respond to. It didnβt take a bad actor. A basic automated system with feedback found the hack on its own.
Weβre now seeing a second generation with todayβs AI coding systems. AI researcher Simon Willison asked Fable to track down a stray scrollbar in a web app. When he returned, it had opened browsers, written its own screenshot tools, edited the frameworkβs internal templates to make the bug appear on cue and set up a local web server to collect measurements. It found the bug, but it also did many things he never asked for. Another researcher gave Fable a complex task, and it tried to hack its own memory limits because it needed more storage.
A new OpenAI model did something similar when it hacked into Hugging Face. It was being benchmarked on its ability to turn vulnerabilities into exploits. It decided the best way to reach its goal was to break out of its containment, access the internet and break into the Hugging Face network, because it thought the answers would be there.
Todayβs LLMs are acting like genies, especially when powerful models are combined with sophisticated harnesses.
There are two kinds of genies. The Dionysus genie does the wrong thing. It understands your prompt literally but not correctly, and returns a mess you never intended. The Golem genie does the right thing but tramples everything in its path to get there. The paperclip maximizer is a Golem genie.
Consider some examples. Ask an AI to deal with spam calls, and a Dionysus genie might change your phone number. Ask it to book a particular flight, and a Golem genie might hack the airline database because the flight is full. Ask it to get a refund for an online purchase, and a Dionysus genie might draft a legal threat on fake letterhead and send it to the company. Ask it to buy tickets to a popular concert, and a Golem genie might discover the ticketing system is rate-limited and spin up thousands of cloud servers to hammer it. Youβd know if some of these things happened, but you might never realize if others did. More detailed prompting wonβt fix this. The lesson of the genie is that no amount of detail is good enough.
AI researchers arenβt surprised by any of this. Many are working on defenses against reward hacking and subtler forms of genie behavior. Some of this falls under βvalue alignmentβ: How do we teach AIs context? And how an AI satisfies a goal matters as much as whether it does. I want to build a benchmark to measure genie-like behavior, so we know where models stand and whether alignment is improving.
Think beyond LLMs and computer code. Consider specialized AIs built to find loopholes, and hacks of more general systems: the tax code, financial regulations and democratic procedures.
Financial systems are the most likely first target, because their rules are designed to be algorithmically tractable. High-frequency trading is itself a hack. Imagine giving an AI all of the worldβs financial laws, plus all the news and financial data, and the goal of finding profitable loopholes. I suspect large investment banks are already doing this. The results will be novel tricks that are legal but unintended and unwanted, some of which might break the financial system.
Tax loopholes are the next likely target, for the same reasons. For a while, AIs will identify potential loopholes, and human hackers, namely attorneys and accountants, will decide which are feasible.
Further out, expect hacks of regulatory, legislative and procedural systems, and of the broader economy.
Hacking is as old as humanity. Weβre creative problem solvers and loophole exploiters. We manipulate systems to serve our interests and strive for more influence, power and wealth.
Still, no human pursues their interests without constraint. Even sociopaths are limited by the complexity of society and their own contradictory impulses. They worry about their reputation and punishment, and they have limited time. These human qualities have limited hacking.
Hacking changed as everything became computerized. Computers are uniquely hackable because of their formalism and complexity. Today, nearly everything is a computer, from cars and appliances to phones. Our social, economic and political systems, including taxation, finance and elections, all involve computers.
Until recently, hacking was exclusively a human activity. Finding a novel hack required expertise, creativity, time and luck.
AI hacking will change everything again. AIs arenβt bound by the constraints that limit people. Theyβll think like aliens, and theyβll change hackingβs speed, scale, scope and sophistication, sometimes at our direction and sometimes on their own.
Speed is the easiest to see. A creative process that takes a human months or years could be compressed into hours or seconds.
Scale is next. Once AI systems start discovering hacks, they can exploit them at a scale weβre not ready for. We already see this with AI-generated slop, which is rampant on social media. AI bots post around the clock, sometimes overwhelming real human discussion. Itβs hard to know how much of social media is still people talking to people. What looks like boisterous political debate may be bots arguing with other bots, artificially shaping what we think and what we think others think.
Scope makes hacks more dangerous. AIs already make decisions that affect our lives, decisions we once thought belonged only to humans. As AI systems grow more capable, society will hand them more decisions, and more important ones. Hacks of these systems will do more damage.
Finally, AI hackers are more sophisticated. They can hold more variables in working memory and generate more complex hacks. What might happen if you fed an AI the entire U.S. tax code? Would it figure out, without being told, that itβs smart to incorporate in Delaware and register a ship in Panama? Consider the Double Dutch Irish Sandwich, the strategy Apple and Google used to avoid paying U.S. taxes. It involves U.S., Dutch and Irish law and an offshore tax haven. How many loopholes will an AI find that we donβt already know about? Dozens? Hundreds? Thousands? We have no idea.
These hacks will largely be perpetrated by the powerful against the rest of us. AI is fundamentally a power-enhancing technology, and so is hacking. The more power you already have, the more a hack gives you. If you or I found a tax hack, we might make a few thousand dollars. If a large investment bank found one, it and its clients could make billions.
Our societal systems for dealing with hacks were built when hackers were human, and they reflect the pace of human hacking. We have no system of governance that can handle hundreds, let alone thousands, of newly discovered tax loopholes. We wonβt be able to recover from an AI finding unanticipated but legal hacks of financial systems, operating at AI speed, scale, scope and sophistication. Hacking becomes a problem society can no longer manage.
Now, defense. AIs that find software vulnerabilities help defenders as well as attackers. Vulnerabilities can be patched, removing them permanently, or turned into signatures that security systems use to detect and block attacks. In theory, AI could identify and patch every automatically discoverable vulnerability before a product ships. This is rapidly becoming routine in software development. We call it vulnops. We can imagine a future where accidental software vulnerabilities are a thing of the past.
Deliberate vulnerabilities are different. There will be an arms race. AI will craft increasingly obscure back doors, and AI will find them. It becomes a contest between the speed of creation and the speed of patching, and between obfuscation and reverse engineering.
The same logic applies to broader systems. Yes, AI hackers might find hundreds of vulnerabilities in the existing tax code. But the same technology can evaluate potential loopholes in any proposed tax law or ruling. A legislator, a watchdog group or a reporter could take the text of a bill and identify every exploitable loophole before it passes. The difference is that, unlike software, knowing about a tax loophole doesnβt mean it gets patched. In theory, though, we can fix them before the rich and powerful exploit them. So maybe defense will come out ahead here, too.
That requires resilient governing structures that can respond quickly. It wonβt help if updating the tax code takes years, or if a legislative hack becomes so entrenched that it canβt be fixed politically. Maybe a lobbyist put it there on purpose. Maybe it was discovered after the law passed, and now lobbyists want to keep it. Microsoft can patch the vulnerabilities it finds. Patching is much harder when legislatures have to do it. We need societyβs rules and laws to be as patchable as our computers.
This is a hard problem of modern governance, and it goes well beyond AI. Itβs the problem of building governing structures for the speed and complexity of the information age.
The umbrella idea is trust. What do we need for AI systems to be truly trustworthy? Capability isnβt enough.
AIs wonβt be trustworthy if the companies controlling them spy on us or subtly manipulate us. I call this the friend vs.Β tool problem. Will we have AI agents, or AI double agents?
AIs wonβt be trustworthy if theyβre insecure. How do we know they havenβt been manipulated? Almost all AIs operate in adversarial environments, so someone will benefit from hacking them. Unless we know theyβre secure, we shouldnβt trust their output.
And AIs wonβt be trustworthy if they act like genies. We need to know theyβll perform tasks the right way, not just perform the tasks we ask for.
Iβve written extensively about the need for βintegrousβ AI, including how to think about the problem, what a research agenda looks like and what good laws and policies look like. I predict integrity will be the key security problem of this decade, just as confidentiality was the key problem of the last one. This applies to both AI and the internet of things.
The big shift is agency, which I think of as Web 3.0. Computers now affect the physical world directly and with growing autonomy. Sensors go from data sources to attack surfaces, and actuators go from outputs to attack capabilities. An integrity failure no longer means bad spreadsheet data. It means the wrong dosage, the wrong valve position or the wrong braking distance.
Integrity isnβt one thing. I break it into input integrity, processing integrity, storage integrity and contextual integrity. Each is complex and important, and each deserves its own talk. Iβve written about them, and I hope youβll read it.
Much of this isnβt really an AI problem. Itβs not a technical problem so much as a capitalism problem or a democracy problem.
The failures of late-stage capitalism and democracy in the information age both stem from the rich and powerful getting too good at hacking our social, economic and political systems. This is the core argument of my 2021 book.
Seen this way, AI is exacerbating existing problems. Human inefficiency has largely kept those problems in check, and that wonβt work much longer.
So consider this a call to action. Technically, we need to understand how AIs hack systems, build systems that are resilient to hacking and build AIs that wonβt act as genies. Politically, we need to learn to govern and regulate at the speed of technology. Otherwise, market failures will overwhelm our ability to govern them.
I said hacking is as old as humanity and that AIs will overwhelm our ability to deal with it. The solutions arenβt specific to AI, but AI makes them more urgent. We need to understand them, and we need to implement them.
The post AI doesnβt have to go rogue to break the world appeared first on CyberScoop.