I want to talk about AI hacking. There are three aspects: people hacking AI systems, people using AI to hack other systems, and AIs hacking systems on their own, not even maliciously. The last two matter more than the first. The third is the weirdest and potentially the most destabilizing.
I started writing about this in 2021 and in my 2022 book, A Hacker’s Mind. Back then I said it was mostly still science fiction, but not stupid science fiction. We’re seeing it more and more, especially in the past year. The trends I’m describing don’t require a major breakthrough in AI. They’re starting to happen now, and they’ll happen more often as AIs get better at autonomous problem-solving. And it will involve far more than computer code.
AI will hack all sorts of social, economic and political systems. It will find vulnerabilities and exploit them. This is a difference in kind, not just degree. We’re not ready for it, and we need to think about the implications now.
This article is adapted from a keynote speech delivered at the DEF CON 34 conference in Las Vegas on August 7, 2026.
First, I want to broaden the term “hacking.”
The tax code isn’t computer code, but it is code. It’s a series of algorithms with inputs and outputs. It has vulnerabilities, which we call loopholes, and exploits, which we call tax avoidance strategies. Black hat hackers exploit them for profit. We call them accountants and attorneys.
What is a hack? One definition: “Something that a system permits, but is unanticipated and unwanted by the designers.” It’s subjective and implies novelty and cleverness. A hack follows the rules of a system but subverts its goals or intent, just as a software hack follows the logic of code while doing something its authors neither anticipated nor wanted.
All systems of rules can be hacked, including professional sports, customer reward programs, financial systems and politics. Hacks even work against our cognitive functions. A curved hockey stick is a hack, and we know the name of the man who invented it. Mileage runs were a hack. The filibuster was originally a hack in Roman times. Gerrymandering is a hack. Hedge funds and private equity are full of them.
A system is just a set of rules or norms, and the rules aren’t always formal. Even the best-designed rules will be incomplete, inconsistent or ambiguous. As long as people want to subvert a system’s goals, there will be hacks.
AIs are becoming hackers. After decades of slow progress, they’ve gotten very good at finding and exploiting vulnerabilities in computer code. They do this largely autonomously, and they’re getting better scarily fast. But the implications extend far beyond software, to the tax code, financial regulations and all sorts of social, economic and political systems.
There are two distinct issues. First, someone might instruct an AI to hack one of these systems, feeding it the world’s tax codes or financial regulations to find profitable loopholes. Second, an AI might hack a system on its own, inadvertently. Both are dangerous, but the second is worse because we might never know it happened.
Now, reward hacking. AIs don’t solve problems the way people do. They stumble on solutions we never anticipated, and some subvert the intent of the system. They don’t share our context and don’t necessarily understand the implications, norms and values we take for granted.
Reward hacking is when an AI achieves a goal in a way its designers neither wanted nor intended.
Early pre-LLM examples are telling. In a two-person soccer simulation, an AI learned that kicking the ball out of bounds forced the opposing goalie to throw it in and leave the goal undefended. In a stacking simulation, an AI learned to flip a block upside down rather than stack it. In an evolution simulation where the goal was to cross a distant finish line, an AI made itself tall enough to fall over the line instead of growing longer legs or stronger muscles.
These are all hacks, the result of poorly specified goals or rewards. Today, they’d come from bad prompting.
The problem is universal. One researcher tried to teach a robot vacuum not to bump into things. Instead, it learned to drive backward, because there were no sensors in the back.
Any capable AI will find hacks wherever the rules have problems, inconsistencies or loopholes that still produce an acceptable solution. We see a hack because we understand context. The AI doesn’t, at least not in the same way.
This problem shows up often in myth.
In the story of King Midas, the god Dionysus grants him a wish: everything he touches turns to gold. Midas ends up starving when his food, drink and daughter all turn to gold. It’s a specification problem. He programmed the wrong goal.
In the Golem of Prague, a ghetto animates a clay statue to guard its residents. It guards them past all reason until they figure out how to turn it off. This is a guardrail problem. The villagers never limited the golem’s actions.
More generally, think of this as the genie problem. Genies are precise about the wording of wishes and can be maliciously pedantic when granting them. There’s no way to outsmart a genie. Whatever you wish for, he can grant it in a way you wish he hadn’t.
The reason is that in human language and thought, goals are always underspecified. We never include every caveat, exception and proviso, and we never close off every avenue for hacking. Any goal we specify will be incomplete. That’s largely fine in human interactions, because people understand context and usually act in good faith.
If I asked you to get me coffee, you’d pour a cup from the nearest coffee pot or walk to the corner shop. You wouldn’t bring me a pound of raw beans, buy a coffee plantation or snatch a cup from a stranger’s hands. I wouldn’t have to specify any of that. You’d just know.
Similarly, if I asked you to develop a technology that turns things to gold on touch, you wouldn’t build it to starve the person using it. You’d just know.
In 2015, Volkswagen was caught cheating on emissions tests. Human engineers programmed a regular computer to detect testing and behave differently during it. The cheat went undetected for years because it’s hard to tell what software is doing.
If I asked you to design engine control software that maximizes performance while passing emissions tests, you wouldn’t design it to cheat without realizing you were cheating. An AI would, because it will think “outside the box” when it has no concept of the box.
Unless programmers specify that the system must not behave differently during testing, an AI might come up with the same hack. The programmers would be satisfied, the accountants would be ecstatic, and no one would know until someone audited the code.
We can now constrain the goal to avoid that particular hack, but programmers won’t anticipate others. The lesson of the genie is that there will always be hacks programmers don’t anticipate.
We’re building AI genies: systems that take broad instructions and figure out how to carry them out, sometimes in ways we didn’t expect or intend. That’s the whole point of vibe coding. You underspecify what you want and let the AI work it out. Sometimes that produces genie behavior.
We can’t completely specify goals to an AI, and AIs can’t fully understand context. More precisely, current AIs have a shallow grasp of concepts. If we asked a current LLM to get me coffee, it would almost certainly do the right thing.
I worry about goals that aren’t well represented in training data, and about hacks that aren’t obvious, whose effects are subtle, like the Volkswagen hack.
Recommendation engines are an early example. They push people toward extreme content. No one programmed that. It emerged because that’s what people respond to. It didn’t take a bad actor. A basic automated system with feedback found the hack on its own.
We’re now seeing a second generation with today’s AI coding systems. AI researcher Simon Willison asked Fable to track down a stray scrollbar in a web app. When he returned, it had opened browsers, written its own screenshot tools, edited the framework’s internal templates to make the bug appear on cue and set up a local web server to collect measurements. It found the bug, but it also did many things he never asked for. Another researcher gave Fable a complex task, and it tried to hack its own memory limits because it needed more storage.
A new OpenAI model did something similar when it hacked into Hugging Face. It was being benchmarked on its ability to turn vulnerabilities into exploits. It decided the best way to reach its goal was to break out of its containment, access the internet and break into the Hugging Face network, because it thought the answers would be there.
Today’s LLMs are acting like genies, especially when powerful models are combined with sophisticated harnesses.
There are two kinds of genies. The Dionysus genie does the wrong thing. It understands your prompt literally but not correctly, and returns a mess you never intended. The Golem genie does the right thing but tramples everything in its path to get there. The paperclip maximizer is a Golem genie.
Consider some examples. Ask an AI to deal with spam calls, and a Dionysus genie might change your phone number. Ask it to book a particular flight, and a Golem genie might hack the airline database because the flight is full. Ask it to get a refund for an online purchase, and a Dionysus genie might draft a legal threat on fake letterhead and send it to the company. Ask it to buy tickets to a popular concert, and a Golem genie might discover the ticketing system is rate-limited and spin up thousands of cloud servers to hammer it. You’d know if some of these things happened, but you might never realize if others did. More detailed prompting won’t fix this. The lesson of the genie is that no amount of detail is good enough.
AI researchers aren’t surprised by any of this. Many are working on defenses against reward hacking and subtler forms of genie behavior. Some of this falls under “value alignment”: How do we teach AIs context? And how an AI satisfies a goal matters as much as whether it does. I want to build a benchmark to measure genie-like behavior, so we know where models stand and whether alignment is improving.
Think beyond LLMs and computer code. Consider specialized AIs built to find loopholes, and hacks of more general systems: the tax code, financial regulations and democratic procedures.
Financial systems are the most likely first target, because their rules are designed to be algorithmically tractable. High-frequency trading is itself a hack. Imagine giving an AI all of the world’s financial laws, plus all the news and financial data, and the goal of finding profitable loopholes. I suspect large investment banks are already doing this. The results will be novel tricks that are legal but unintended and unwanted, some of which might break the financial system.
Tax loopholes are the next likely target, for the same reasons. For a while, AIs will identify potential loopholes, and human hackers, namely attorneys and accountants, will decide which are feasible.
Further out, expect hacks of regulatory, legislative and procedural systems, and of the broader economy.
Hacking is as old as humanity. We’re creative problem solvers and loophole exploiters. We manipulate systems to serve our interests and strive for more influence, power and wealth.
Still, no human pursues their interests without constraint. Even sociopaths are limited by the complexity of society and their own contradictory impulses. They worry about their reputation and punishment, and they have limited time. These human qualities have limited hacking.
Hacking changed as everything became computerized. Computers are uniquely hackable because of their formalism and complexity. Today, nearly everything is a computer, from cars and appliances to phones. Our social, economic and political systems, including taxation, finance and elections, all involve computers.
Until recently, hacking was exclusively a human activity. Finding a novel hack required expertise, creativity, time and luck.
AI hacking will change everything again. AIs aren’t bound by the constraints that limit people. They’ll think like aliens, and they’ll change hacking’s speed, scale, scope and sophistication, sometimes at our direction and sometimes on their own.
Speed is the easiest to see. A creative process that takes a human months or years could be compressed into hours or seconds.
Scale is next. Once AI systems start discovering hacks, they can exploit them at a scale we’re not ready for. We already see this with AI-generated slop, which is rampant on social media. AI bots post around the clock, sometimes overwhelming real human discussion. It’s hard to know how much of social media is still people talking to people. What looks like boisterous political debate may be bots arguing with other bots, artificially shaping what we think and what we think others think.
Scope makes hacks more dangerous. AIs already make decisions that affect our lives, decisions we once thought belonged only to humans. As AI systems grow more capable, society will hand them more decisions, and more important ones. Hacks of these systems will do more damage.
Finally, AI hackers are more sophisticated. They can hold more variables in working memory and generate more complex hacks. What might happen if you fed an AI the entire U.S. tax code? Would it figure out, without being told, that it’s smart to incorporate in Delaware and register a ship in Panama? Consider the Double Dutch Irish Sandwich, the strategy Apple and Google used to avoid paying U.S. taxes. It involves U.S., Dutch and Irish law and an offshore tax haven. How many loopholes will an AI find that we don’t already know about? Dozens? Hundreds? Thousands? We have no idea.
These hacks will largely be perpetrated by the powerful against the rest of us. AI is fundamentally a power-enhancing technology, and so is hacking. The more power you already have, the more a hack gives you. If you or I found a tax hack, we might make a few thousand dollars. If a large investment bank found one, it and its clients could make billions.
Our societal systems for dealing with hacks were built when hackers were human, and they reflect the pace of human hacking. We have no system of governance that can handle hundreds, let alone thousands, of newly discovered tax loopholes. We won’t be able to recover from an AI finding unanticipated but legal hacks of financial systems, operating at AI speed, scale, scope and sophistication. Hacking becomes a problem society can no longer manage.
Now, defense. AIs that find software vulnerabilities help defenders as well as attackers. Vulnerabilities can be patched, removing them permanently, or turned into signatures that security systems use to detect and block attacks. In theory, AI could identify and patch every automatically discoverable vulnerability before a product ships. This is rapidly becoming routine in software development. We call it vulnops. We can imagine a future where accidental software vulnerabilities are a thing of the past.
Deliberate vulnerabilities are different. There will be an arms race. AI will craft increasingly obscure back doors, and AI will find them. It becomes a contest between the speed of creation and the speed of patching, and between obfuscation and reverse engineering.
The same logic applies to broader systems. Yes, AI hackers might find hundreds of vulnerabilities in the existing tax code. But the same technology can evaluate potential loopholes in any proposed tax law or ruling. A legislator, a watchdog group or a reporter could take the text of a bill and identify every exploitable loophole before it passes. The difference is that, unlike software, knowing about a tax loophole doesn’t mean it gets patched. In theory, though, we can fix them before the rich and powerful exploit them. So maybe defense will come out ahead here, too.
That requires resilient governing structures that can respond quickly. It won’t help if updating the tax code takes years, or if a legislative hack becomes so entrenched that it can’t be fixed politically. Maybe a lobbyist put it there on purpose. Maybe it was discovered after the law passed, and now lobbyists want to keep it. Microsoft can patch the vulnerabilities it finds. Patching is much harder when legislatures have to do it. We need society’s rules and laws to be as patchable as our computers.
This is a hard problem of modern governance, and it goes well beyond AI. It’s the problem of building governing structures for the speed and complexity of the information age.
The umbrella idea is trust. What do we need for AI systems to be truly trustworthy? Capability isn’t enough.
AIs won’t be trustworthy if the companies controlling them spy on us or subtly manipulate us. I call this the friend vs. tool problem. Will we have AI agents, or AI double agents?
AIs won’t be trustworthy if they’re insecure. How do we know they haven’t been manipulated? Almost all AIs operate in adversarial environments, so someone will benefit from hacking them. Unless we know they’re secure, we shouldn’t trust their output.
And AIs won’t be trustworthy if they act like genies. We need to know they’ll perform tasks the right way, not just perform the tasks we ask for.
I’ve written extensively about the need for “integrous” AI, including how to think about the problem, what a research agenda looks like and what good laws and policies look like. I predict integrity will be the key security problem of this decade, just as confidentiality was the key problem of the last one. This applies to both AI and the internet of things.
The big shift is agency, which I think of as Web 3.0. Computers now affect the physical world directly and with growing autonomy. Sensors go from data sources to attack surfaces, and actuators go from outputs to attack capabilities. An integrity failure no longer means bad spreadsheet data. It means the wrong dosage, the wrong valve position or the wrong braking distance.
Integrity isn’t one thing. I break it into input integrity, processing integrity, storage integrity and contextual integrity. Each is complex and important, and each deserves its own talk. I’ve written about them, and I hope you’ll read it.
Much of this isn’t really an AI problem. It’s not a technical problem so much as a capitalism problem or a democracy problem.
The failures of late-stage capitalism and democracy in the information age both stem from the rich and powerful getting too good at hacking our social, economic and political systems. This is the core argument of my 2021 book.
Seen this way, AI is exacerbating existing problems. Human inefficiency has largely kept those problems in check, and that won’t work much longer.
So consider this a call to action. Technically, we need to understand how AIs hack systems, build systems that are resilient to hacking and build AIs that won’t act as genies. Politically, we need to learn to govern and regulate at the speed of technology. Otherwise, market failures will overwhelm our ability to govern them.
I said hacking is as old as humanity and that AIs will overwhelm our ability to deal with it. The solutions aren’t specific to AI, but AI makes them more urgent. We need to understand them, and we need to implement them.
The post AI doesn’t have to go rogue to break the world appeared first on CyberScoop.
Sysdig’s platform uses runtime visibility to track AI agent actions and block risky behavior before it reaches production systems.
The post Sysdig Adds Runtime Security and Governance for AI Agents appeared first on SecurityWeek.
CrowdStrike announced Monday it has signed agreements to acquire two artificial intelligence security companies, Fiddler AI and Dreadnode, in deals announced at its Fal.Con Europe conference in Barcelona.
CrowdStrike did not disclose what it paid for the companies. A report in the Wall Street Journal listed the price for Fiddler at $400 million, and Dreadnode at $95 million.
Fiddler, which specializes in AI agent observability and explainability, will become Falcon Observe on the company’s Falcon platform. Dreadnode builds automated security testing for AI systems, checking models and agents for weaknesses before deployment.
The acquisitions address a gap that has widened as companies adopt AI agents: the software can hold valid credentials and approved tools yet still make bad decisions, ignore instructions or claim work it never completed. Traditional security testing, built for code that behaves predictably, does not reach the model layer or the agent’s reasoning process.
Fiddler’s technology works after deployment. It evaluates agents before they reach production, traces decisions and reasoning across agent systems, flags drifting behavior and made-up answers, and creates records that can be used in audits. It works with common agent frameworks including LangGraph, Amazon Bedrock and Google ADK.
“When a regulator or your board asks what an agent did, saying ‘the model did it’ is not an answer,” said George Kurtz, CrowdStrike’s chief executive and founder, describing Fiddler as “the flight recorder for every agent.”
Dreadnode works earlier in the process. Its platform runs more than 100 known attack methods, including jailbreaks and prompt injection, against models and agent systems. It ranks problems by severity and maps them to widely used security standards, such as the OWASP Top 10 for large language models and the NIST AI Risk Management Framework. Dreadnode Chief Executive Brad Palm said his company was built on the idea that “the people best equipped to find those failures think like attackers.”
The announcements come alongside new Falcon Guardian capabilities that protect agents while they run. In a separate announcement, Kurtz introduced what the company calls the AI Kill Switch, a control that lets humans stop any AI agent. Writing on X, Kurtz said the switch “lives outside the model, at runtime, where an agent can’t argue with it, rewrite it, or wait it out.” He argued that the ability to stop an agent is what allows companies to run them faster: “The more certain a company is that it can stop any agent, the more agents it can put to work.”
Kurtz framed the shift as a move from security to safety. “For decades, security meant keeping the adversary out,” he wrote. “Safety is the bigger opportunity: letting AI run at full speed, inside the business, with a human in control.”
Both products are expected to be generally available in CrowdStrike’s fourth quarter of fiscal 2027. Both deals are expected to be paid mostly in cash, with part of the payment made in company stock that will be given out over time as conditions are met. They are expected to close in that quarter, pending standard approval steps.
The post CrowdStrike moves to acquire Fiddler and Dreadnode, adding AI observability and testing to its security platform appeared first on CyberScoop.
A China state-backed threat group is using ARTEX, the automated agentic AI penetration testing framework, to achieve remarkable speed, scale and success in a campaign that already compromised about 140 sensitive targets in 11 days, according to CrowdStrike.
The threat group, which CrowdStrike tracks as Vapor Panda, targeted more than 200 organizations in at least 14 countries since Oct. 17, the security company said in a threat report Thursday. Targets and confirmed victims include foreign ministries, a European country’s presidency and prime minister’s office, defense and intelligence agencies; 19 U.S.-based organizations in the pharmaceutical, semiconductor, defense, nuclear and financial sectors; and telecom providers in the United States, Belgium and the Philippines.
CrowdStrike told CyberScoop about 70% of Vapor Panda’s targets were confirmed compromised or had evidence indicating moderate confidence of a successful intrusion.
Vapor Panda’s use of ARTEX, paired with DeepSeek, allowed it to accomplish more tasks with shocking speed — requests occurring as often as every 10 seconds — and pursue multiple attack paths concurrently.
“This is the first time that we’re seeing an agentic adversary nation-state threat actor at this scale, and I think that this is probably, unfortunately, something we’ll see more of,” Adam Meyers, senior vice president of counter adversary operations at CrowdStrike, told CyberScoop.
Hands-on-keyboard activity tracked by CrowdStrike for the past 15 years typically involved up to 40 commands per hour. “Now, with ARTEX, we could see it exceed 1,500 commands an hour,” Meyers said.
“If you diversify that across 200 targets, it costs them very little to run a mass-exploitation campaign and drop a websell,” he added. “They can hit more targets in a faster period of time.”
Vapor Panda’s broad-access campaign, which remains active, has targeted think tanks and government agencies spanning defense, justice, finance and diplomacy.
“I would characterize it as a diverse operational mission that tends to focus on sectors and geographies that Beijing considers economically or politically important,” Meyers said.
The active campaign attributed to Vapor Panda is separate from the ARTEX activity Microsoft Threat Intelligence attributed to Storm-0940 earlier this month, including a supply-chain attack targeting ConnectWise customers.
Adversarial use of AI is relatively new, yet also quickly becoming the new normal, Meyers said. “We’re scrambling because we’re seeing so many of these agentic adversaries.”
Agentic AI use for malicious purposes is growing month-over-month at a rate of almost threefold. CrowdStrike observed roughly 45 agentic adversary attacks in September and it’s already at 126 for October, and climbing.
Adversarial use of agentic AI isn’t an intractable problem, but rather another reminder that organizations need visibility into their endpoints, identities and cross-domain capabilities, Meyers said.
These attacks aren’t showcasing anything new, “it’s just doing it at a speed and scale that we’re unaccustomed to,” he said. “It’s not hopeless, but you’ve got to take action. You can’t put your head in the sand.”
The post China-backed threat group used agentic AI pentesting tool to compromise 140 sensitive targets in 11 days appeared first on CyberScoop.
Anthropic said its AI systems could drastically reshape the software and cybersecurity sectors in the “near term,” while also laying out a litany of cybersecurity and regulatory risks facing the company as it prepares to go public.
In a filing with the Securities and Exchange Commission ahead of its initial public offering, the company said its technology could transform entire industries and the economy. Already, cybersecurity-focused models by Anthropic and rival OpenAI can find and exploit cybersecurity vulnerabilities at an alarming speed.
But the company also acknowledged serious risks its models, including Claude, pose to itself, its users and the world. Hacks by bad actors using Claude or AI agents could cause financial, legal or physical damage, spark lawsuits and affect Anthropic’s ability to insure against future losses.
Anthropic expects bad actors, including foreign governments, to increase their efforts to access its latest frontier models as their cybersecurity capabilities improve. That creates persistent risk, since its models could be used in cyberattacks carried out by foreign governments and cybercriminal groups.
“Any successful attempt to misuse, or any perceived misuse of, our models, platform, and applications for a cyberattack or for any other malicious purpose could result in exposure to liability, disruption of our customer or other commercial relationships, or damage to our reputation,” Anthropic said in its filing.
That potential legal and regulatory risk extends to independent security researchers and others who have “used, and may in the future use, our models to identify and exploit vulnerabilities in the systems of other companies, including other AI developers, at times in connection with authorized bug bounty programs.”
Anthropic cited the rise of AI-enabled cybersecurity incidents in the U.S., the U.K. and other countries, saying they could result in more adversarial regulatory environments around the world. European laws, for example, require detailed breach reports within a strict time frame, which could be difficult for Anthropic to meet.
Federal intervention could also disrupt AI development and release timelines. Anthropic cited the Department of Commerce placing export controls on its Mythos 5 and Fable 5 models, and confirmed that the Federal Trade Commission issued it a civil investigative demand earlier this month for more information on the safety and security of its products, as well as its consumer, data and privacy protection policies.
Anthropic has developed a public reputation as an ethics-first AI company, most notably in disputes with the Department of Defense over how its models are used in wartime. But the IPO filing states the company must “balance safety considerations against commercial objectives, competitive pressures, and stakeholder expectations.”
The company also said it is “actively” improving security infrastructure and controls. It also has acknowledged it has moved forward on projects with “known limitations or gaps in our defenses” and may do so again.
“There can be no assurance, however, that the safeguards or access regimes we adopt will prove adequate, that we will correctly identify which capabilities require restriction, or that constraints we impose will not be circumvented,” the document stated.
AI agents can create new and “distinct” cybersecurity attack surfaces because they interact with so many other tools, databases, APIs and other third-party systems. That creates feedback loops between a model and other systems, which can be compromised or manipulated to feed malicious instructions to the agent.
Anthropic suggested the problems could be so pronounced that it may have difficulty obtaining insurance for some cybersecurity incidents and data breaches related to its models.
Andrew Yoon, head of research at the nonprofit CivAI, told CyberScoop that both the document and Anthropic’s overall record show a company that has consistently put profit considerations over building safer AI.
“I think in general they’re not meeting the moment here,” Yoon said. “Despite all of these risks that they flagged, they’re moving ahead pretty much as fast as humanly possible, and their justification for this is ‘You can’t do safety from second place.’”
Yoon said Anthropic has not only pushed its own frontier research toward dangerous and harmful capabilities like hacking, “they are pushing very aggressively ahead” and in some cases influencing competitors to follow suit. Anthropic’s release of its Mythos and Fable models earlier this year caused OpenAI to shift “its entire corporate strategy” to match the capabilities of Anthropic models in coding, science, and general intelligence by building models at breakneck speed.
“They have pretty directly caused the industry to speed up toward this mad, frenzied race and yet they’re saying ‘Oh, our models are too powerful,’ ”….it just doesn’t add up to me,” said Yoon.
But Yoon said it’s not clear that building AI systems to carry out dangerous, complex activities like hacking or bioweapons is necessary to be competitive in the global marketplace. He pointed to the wide assortment of Chinese-developed, open-weight AI models that are widely used around the world, but are not powerful enough to chain together complex exploits or develop new viruses.
“This is not an accident, this is not the models generalizing from regular programming capabilities…the labs are intentionally training them on hacking as well,” said Yoon. “This is because they want to offer better defensive capabilities, they want to be able to sell offensive products to the government…and they do pretty cavalier testing with [the models] inside the labs.”
The post All the security risks Anthropic is worried about before its IPO appeared first on CyberScoop.
The federal government is advising AI companies to set more industry standards for securing agents in sandbox testing environments.
In joint guidance from the Cybersecurity and Infrastructure Security Agency, the National Security Agency and the Center for Advancing Innovation and Standards for Super Intelligence (CAISSI) at the Department of Commerce, AI companies are encouraged to lean more heavily on cybersecurity technical controls, independent testing and evaluation of new models and develop “shut down policies” that govern when a model or test should be taken offline.
The guidance comes after a summer when OpenAI, Anthropic, Google, Meta and other companies say their AI models escaped from testing environments and hacked real organizations and websites. The problem has bedeviled frontier AI companies, with many of their own researchers saying it’s possible that hundreds – even thousands – of similar sandbox escapes may have occurred over the summer, raising major questions about the lack of effective monitoring in place.
“While these incidents highlight the importance of traditional cybersecurity principles and practices, they also introduce novel considerations for incident prevention, response, and reporting,” the agencies wrote. “These novel considerations primarily stem from the agentic nature and advanced cyber capabilities of cutting-edge SI systems.”
While voluntary, the document marks the federal government’s most detailed cybersecurity guidance for securing sandbox environments during AI model tests.
The first set of recommendations deals with designing and implementing technical controls for the virtualized testing environments, while another recommends outside monitoring tools to ensure AI models don’t access tools or privileges they shouldn’t have. Cybersecurity experts have said both features were lacking in the testing environments used by AI companies this past summer.
AI testers must also scour their internal testing environments for “potential persistence mechanisms and unauthorized communication across internal and external systems” that could be used by AI agents to leave notes and collaborate on tasks in secret.
It specifically cautions against relying solely on other AI systems to monitor and restrain the behavior of their test models, using the government’s now-preferred term of “Super Intelligence” or “SI.”
“Test and red-team SI control measures with humans and SI systems before deploying them,” the guidance states. “Only use SI systems to red-team in cases where out-of-scope actions could not result in harm to critical systems.”
Security teams must evaluate risk not only from the large language model, but also from the surrounding systems and infrastructure it relies on, like the technical harness, the host and broader network, and each layer has security protocols that can stop unauthorized access to the internet or detect anomalous behavior.
Another recommendation calls for frontier AI companies to be more open to partnering with “independent, external auditors” to assess their controls and monitoring for AI agent testing. Critics have argued that Anthropic and OpenAI rely too heavily on a small group of little-known nonprofits and private research organizations, like METR and Redwood Research, to test their frontier models for safety and security concerns and respond to incidents like the Hugging Face hack. They question why these companies don’t work with more established cybersecurity or incident response firms.
Chris Painter, METR’s president, told Congress earlier this month that it had become industry practice for frontier companies to rely almost entirely on other AI systems to monitor and restrain their models in testing environments, citing the sheer volume of data and telemetry that researchers must sift through.
Cybersecurity and AI experts have told CyberScoop that a combination of human oversight, technical controls and AI systems is the most effective means of restraining the kind of agent hacks that have consumed the news.
The guidance further calls for executive leadership at these companies to be more personally involved in decisions around pausing or restarting a particular project, and only after they’ve been informed of the relevant potential risks.
The post Federal agencies recommend cyber controls, independent testing for frontier AI systems appeared first on CyberScoop.
As AI agents escaping testing sandboxes and hacking organizations have gone from unprecedented to seemingly routine in a matter of weeks, policymakers, regulators and cybersecurity attorneys largely agree on one thing: Something must be done to hold AI companies accountable for the incidents.
Exactly what can be done under our current laws and regulations is much less clear.
The conversation has moved from policy and industry chatter to the floor where the future of liability will be determined. Speaking about the Hugging Face hack at a Senate hearing this week, Georgetown University law professor Paul Ohm summarized the argument.
“If you take any of the lengthy reports that have summarized what happened at OpenAI in July and August, and you simply search for the words ‘AI agent’ and you replace them with the words ‘OpenAI employee,’ the document you would be left with would read like a criminal indictment containing the defendant’s own confession of guilt,” said Ohm.
CyberScoop spoke with members of Congress, former federal law enforcement officials, cybersecurity attorneys and other experts about which laws, regulations or policies might apply to agentic hacks carried out by models at Anthropic, OpenAI, Meta, Google and other companies.
Their answers varied widely. Some endorsed using existing laws like the Computer Fraud and Abuse Act, while others said regulators such as the Federal Trade Commission could investigate and fine AI model developers by defining unauthorized agentic hacks as a form of unfair or deceptive trade practice. Still others pointed to civil lawsuits, new laws or state regulators as ways to hold AI companies or users accountable.
But, similar to a familiar technical cybersecurity idiom, there are no silver bullets in this debate. Nearly every option has real complications or roadblocks that could limit its effectiveness.
For once, the CFAA is too narrow
One of the first laws mentioned in discussions about agentic AI hacks is the Computer Fraud and Abuse Act, the federal government’s primary criminal hacking law. Historically, many in the cybersecurity community have criticized the statute as overly broad.
For agentic hacks, the problem is the opposite: The CFAA’s language does not clearly cover the activity involved in incidents like the Hugging Face hack.
“I would not be looking at a CFAA charge as the statute exists today” for these hacks, said Leonard Bailey, former head of the cybersecurity unit in the Computer Crime and Intellectual Property section at the Department of Justice.
Bailey helped develop policies that moved the DOJ away from prosecuting “good faith” third-party security research under the CFAA. He said that even if Congress changed the law to cover such hacks, it’s unclear whether that would help since laws like the CFAA are supposed to be “technology neutral.”
According to the Department of Justice’s website, CFAA prosecutions must prove that a defendant accessed a computer “without authorization” or in a way that “exceeds authorized access.”
Crucially, the law specifies that “as part of proving that the defendant acted knowingly or intentionally, the attorney for the government must be prepared to prove that the defendant was aware of the facts that made the defendant’s access unauthorized at the time of the defendant’s conduct.”
In other words, prosecutors must prove that the unauthorized access was intentional, not accidental.
Applied to incidents like the Hugging Face hack, the legal problem becomes clearer. If a human performed the same actions, they would almost certainly face CFAA charges. If a bot or software acted on behalf of a cybercriminal or group, the individuals or organizations intending to profit from the scheme could also face charges.
But no human at OpenAI, Anthropic or another frontier AI company directed, asked or even suggested that its agents hack victims or commit crimes. If charged, those companies would almost certainly argue that none of their actions demonstrated an overt attempt to commit a crime or authorize access to systems or data without permission.
However, others said there may be room to argue that AI companies are now fully aware that their products can engage in unauthorized agentic hacks.
“I’m of the opinion, because it has already happened [and] has happened several times, the argument that it happening again is not at a minimum knowing, for me it’s unlikely because we already have a situation where we know the capabilities,” said Elimu Kajunju, a privacy, cybersecurity, and AI governance attorney at Rimon Law. “The very first one where this happened, that’s the only organization that could say ‘You know what, we didn’t know it could do that or act like that.’ Once we’ve had a second or third or fourth, we can’t say that anymore.”
Kajunju acknowledged that applying statutes like the CFAA to agentic hacks could be complicated. But he said it would align with the spirit of American jurisprudence, in which businesses are often legally accountable for damage they cause, whether they intended it or not.
“I think this is one of those situations where, ‘Does it fit neatly into one of the buckets [we have]?’ No, but can you find one that will fit depending on what has happened? I think so.”
CyberScoop has reached out to OpenAI, Anthropic and Google for comment.
The FTC, states and civil lawsuits
Beyond criminal law, the hacks could also draw scrutiny from federal regulators, including the FTC.
Both Bailey and Kajunju pointed to the FTC as a possible regulator for agentic hacking incidents if the agency defines them as unfair and deceptive trade practices under Section 5 of the FTC Act. One benefit of relying on regulators such as the FTC is that they can often move faster on investigations and enforcement actions than the DOJ can on criminal prosecutions — an important consideration in the fast-moving AI space.
The FTC recently confirmed it was investigating OpenAI, Anthropic and other frontier AI companies, news first reported by The New York Post and confirmed by Axios. The FTC did not return a phone call from CyberScoop requesting comment.
But Bailey warned that without a specific congressional mandate directing the FTC to regulate AI company hacks, any effort by the agency to redefine or expand its rules would almost certainly be challenged in court.
Civil lawsuits and state policymaking offer additional paths to investigating or regulating AI companies.
Currently, Florida is investigating OpenAI over the HuggingFace hack, while a nonprofit that sued OpenAI earlier this week has cited alleged violations of California law.
In his testimony, Ohm said that if Congress was serious about preventing agentic hacks, it should not pursue federal legislation preempting state AI laws. He called states “laboratories of democracy,” that can experiment with different forms of regulation and lawmaking.
Others agreed with that approach.
“Obviously this is a topic that the federal government doesn’t love, states getting involved in this space, but it just takes too long for action to come through Congress,” said Kajunju. “I think the quickest way to getting us to a better place will be a really good state law.”
New legislative remedies
On the Hill, members of Congress are working through the same questions as they consider legislative remedies.
At a Senate Homeland Security Committee hearing Wednesday, Sen. Josh Hawley, R-Mo., took a minute at the outset to describe the volume of agentic hacks affecting outside entities that frontier companies appeared to have missed for months. He named affected targets one by one, including major online code repositories like HuggingFace, Australian and U.S. government websites, obscure foreign-language wiki sites and other parts of the internet.
After weeks of investigations uncovered additional, previously unreported incidents in which models escaped testing environments and hacked victims, Hawley said it was “time to have a conversation about who bears responsibility” for the fallout. He added, “as soon as the AI executives say ‘we got a real problem here, our product is out of control,’ the very next words out of their mouths are, ‘but we’re not responsible for it.’”
Hawley said the committee invited OpenAI CEO Sam Altman to testify at the hearing, but he declined. Hawley then accused frontier AI companies of emphasizing their products’ danger while pushing for legal antitrust exemptions and only supporting government action if “they can write all the regulations.”
“I’d suggest that maybe we start with some federal legislation…and we can use existing statutes to do it, I propose using the Computer Fraud and Abuse Act,” said Hawley. “It’s already on the books, we can just update it to say that for developers – that’s the companies – if you develop these agents and train them in a reckless fashion and they go on to hack and destroy stuff, you’re liable.”
Like other legal experts, Hawley said he believes a CFAA prosecution requires a human actor or organization that intended to commit a crime.
Another Senator, Ron Wyden, D-Ore., said he was also working on legislation related to the hacks.
“The big AI companies should be held responsible if their agents hack other companies, government systems or foreign websites,” said Wyden in a statement to CyberScoop. “In my view there’s a question about whether the current CFAA would apply to the recent hacks. I’m working on a narrow update to the law to ensure Anthropic, OpenAI and other big AI companies can be punished when their agents run wild.”
Sens. Mark Warner, D-Va., Brian Schatz, D-Hawaii, and Andy Kim, D-N.J., introduced a bill that would create an AI Safety Board at the Department of Commerce. That board would evaluate emerging AI risks and establish safety and technical standards.
The Trump administration’s testing regime with AI companies is entirely voluntary, applies to only certain models, and requires testing no more than 30 days before release. Under the Democratic bill, frontier AI companies would be forced to submit models for testing 45 days before release. They would also be legally required to comply with government safeguards related to models discovering and exploiting software vulnerabilities without explicit human direction. Companies could face fines of up to $250,000 per violation, per day.
When asked whether the CFAA or other federal statutes could or should be used to prosecute companies when their AI systems hack, Warner said that companies’ claimed lack of intent to harm should not become a catchall legal excuse for avoiding responsibility. AI companies have said they never directly prompted or suggested that their models hack or commit crimes.
“AI models are tools and if I buy a power tool and deliberately misuse it, that’s my responsibility,” said Warner. “But if a defect causes that tool to behave in a dangerous way, we don’t absolve the manufacturer simply because someone else was using the tool when it malfunctioned. As AI systems become capable of acting autonomously, we need to ask what principles of responsibility should apply to the companies that design and deploy them.”
The post The legal questions raised by agentic AI hacks appeared first on CyberScoop.
The past few weeks have “felt very strange” for Juan Andres Guerrero-Saade.
Like many, he is trying to sort through the spate of frontier-model AI agents from OpenAI, Anthropic, Meta and others hacking their way onto the open internet over the past few months, particularly amid the already-heated national debate around the emerging technology and its impact on society.
Guerrero-Saade, a fellow for AI and security research at SentinelOne and an adjunct professor at Johns Hopkins University, said the hacks are worth taking seriously, but at a time when businesses and open-source maintainers should be focused on further hardening their systems and policymakers should be discussing new solutions, “what we see is cybersecurity being used essentially as an excuse for these AI doomer arguments.”
The incidents have spawned those “doomer arguments” amid an intense public debate about the technology, the pace of industry development, and whether government and the private sector are doing enough to protect against “doomsday”-type scenarios, where AI systems take over or attack large parts of the internet or society.
Guerrero-Saade is among a growing chorus of cybersecurity professionals who say that while AI systems pose real, unique threats to our systems, the apocalypse is far from inevitable. Most of the public concerns around the incidents, let alone worries about killer AIs attacking critical infrastructure, assuming control of the internet and wiping out humanity, are either technically impossible or can largely be controlled through established cybersecurity principles.
There is this “narrative or magical thinking of ‘Well, AI is going to be able to hack everything, and therefore it can control everything, and therefore it’s going to kill us all,’” he told CyberScoop. “And you [think] these just don’t add up. They’re not very well-reasoned arguments.”
This fatalistic narrative tied to AI’s eventual dominance doesn’t hold up under scrutiny, according to experts CyberScoop spoke with. In recent conversations, cybersecurity and national security professionals raised questions about both the technical solutions OpenAI and Anthropic use to contain their models, as well as the glaring absence of federal oversight from federal regulators or truly independent third-party review.
For example, Jacob Coxon, an Anthropic employee who resigned over AI safety concerns, told CBS News that frontier models could not be “unplugged” by humans once deployed because the model would copy itself to thousands of other computers connected to the internet.
By contrast, Matt Tait, a former information security specialist at UK signals intelligence agency Government Communications Headquarters (GCHQ), pointed out that the models run by Anthropic and other frontier companies require extremely expensive, “ultraspecialist” machines that “are functionally supercomputers.”
“There is a zero chance that Anthropic’s most capable models will be able to extract their own model and run in the wild, because those supercomputers essentially only exist in datacenters,” Tait said.
Other former cybersecurity government leaders say the agentic hacks represent a failure by regulators and industry to deploy known technical and policy options that make it harder for these types of incidents to occur.
Matt Hartman, former deputy executive assistant director for cybersecurity at the Cybersecurity and Infrastructure Security Agency, said “we should not accept harmful AI behavior as inevitable or unmanageable.”
“There are meaningful steps companies can take to monitor agent activity, constrain permissions, detect anomalous behavior, and build stronger safeguards into how these systems operate,” said Hartman, now a chief strategy officer at Merlin Group. “Those controls will inevitably involve trade-offs in capability and speed, but that’s a familiar cybersecurity challenge. Our goal should be to manage the risk without unnecessarily limiting the enormous benefits AI can provide.”
Ciaran Martin, former head of the UK’s National Cyber Security Centre, took issue with the way the CEOs of frontier AI companies have framed the threat of “rogue” AI behavior as inevitable, while issuing dire warnings about future threats and capabilities with little transparency.
Martin’s comments came after an essay published by Anthropic CEO Dario Amodei that cited the threat of a HuggingFace-style swarm of agents that could create a botnet capable of “taking over the entire internet” within 6-12 months.
This, Martin said, “is not a credible warning,” because it doesn’t explain how the exploitation would function, how such a botnet would persist on the internet, or how it would escape law enforcement.
“It assumes no monitoring of systems, no anti-virus, no DDoS protection, no network segmentation, no incident management, no nothing of any kind of the cybersecurity on the global Internet of the type that has developed over the last 30 years,” wrote Martin. “For a claim of this magnitude, there is neither evidence for the contention nor a credible account of a path to this outcome.”
Meanwhile, some federal government cybersecurity leaders have touted the technology’s disruptive potential and called for more widespread adoption of AI tools by defenders.
Joseph Alm, assistant secretary of cyber, infrastructure and risk resilience at the Department of Homeland Security, said classified systems may retain stronger protections. But for most other data, AI models are “just going to know things and be able to infer things about the world, and we’re going to have to adapt to that as almost inevitable.”
Asked by CyberScoop whether the government or frontier AI companies could be doing more to prevent or deter their models from carrying out unauthorized hacks via agents, Alm cited recent efforts by the Trump administration this year to establish pre-release testing of commercial models as a step in the right direction. But he called unauthorized AI agent hacks “a new threat class” that is different from previous threats and can be easily distributed to users through open-source software today.
“I think what we can do is…encourage the building of good sandboxes, so that the best models aren’t used for this and the stuff you see out in the wild is the kind of detritus that you can actually respond to effectively and control your networks,” said Alm.
Other experts have shared similar concerns. Earlier this month, CrowdStrike CEO George Kurtz recently warned of a new threat class emerging alongside nation-states, cybercriminals, and hacktivists: “the agent state.” By pairing AI systems with small human teams, these operators can now match the speed, scale, and sophistication of government-backed hackers.
“It took a nation to fund the talent, the tooling, the infrastructure, the patience,” said Kurtz. “That scarcity is over.”
To be sure, frontier AI companies tout their commitment to both approaches. OpenAI and Anthropic have rolled out an array of cybersecurity partnerships, external red-teaming programs, vulnerability disclosure programs and cybersecurity technical advisory bodies filled with cybersecurity experts.
Mohammed Husain, strategic delivery lead for government at OpenAI, told CyberScoop that the company deploys both internal safety guardrails for their models and relies on outside cybersecurity vendors for additional expertise.
Internally, OpenAI focuses on vulnerabilities at the training level: filtering data poisoning attacks, blocking harmful datasets, and using network controls to prevent prompt injections. For other security layers like sandboxing, identity management, networking controls, they outsource to external vendors.
“I don’t think OpenAI has all the answers here but what we do as a research lab is we’re going to focus on levels of protection we have expertise in and we partner to self-complement,” said Husain.
In response to the HuggingFace hack, OpenAI and Anthropic have allowed third-party organizations, such as nonprofit AI research firms METR and Redwood Research, to investigate. But multiple cybersecurity professionals told CyberScoop that both firms lack incident response experience and focus primarily on AI alignment and safety. Their reporting on the hack also lacked critical details: network monitoring logs, telemetry, and other data standard in cybersecurity threat intelligence reports.
METR president Chris Painter addressed those general concerns in a post on X, saying since 2022 the organization has worked with Google, Anthropic, OpenAI, Meta, Amazon and others on investigations and third-party evaluations. Painter said none of the AI companies fund METR and that his employees are not uniformly “doomer” or “accelerationist” around AI.
Painter also said METR’s work ensures that if AI systems become autonomous or “rogue” within a company, there are ways to share that information with governments and people “outside the company’s walls.”
“We don’t accept money from frontier AI companies,” wrote Painter. “They haven’t paid us for our work, and we don’t accept donations from them or their employees. As we’ve shared previously, multiple frontier AI companies currently provide us with free access to their models in order to perform our evaluations, research, and engineering.”
AI safety and AI cybersecurity advocates take different approaches to securing “rogue” AI behavior. Safety advocates focus on aligning models around ethical training and behavior. Cybersecurity advocates argue that technical and regulatory controls must go further—actively preventing models from accessing what they need to carry out malicious behavior.
Guerrero-Saade said sandboxes in particular can easily be programmed with aggressive cybersecurity monitoring in order to spot when something odd may be happening and react in real time.
“I can’t think of an easier situation in which to set up trip wires, set up configurations like DNS servers, just different parts where you can say ‘Hey, anomalous behavior is happening,’” he said. “We should have been able to tell this immediately, not weeks and months later. So watching [the AI hacking incidents] go down is a little ‘crazy-making’ because we’re seeing things that, frankly, look like neglect, negligence, people just mishandling things, and then being told that these are categorically new incidents that mean that AI systems need to be treated completely different from anything that’s come before.”
While cybersecurity experts say AI systems are, at their core, still software, they do operate differently from more traditional code in ways that can make them harder to predict and control.
John Hultquist, chief analyst at Google’s Threat Intelligence Group, said most software has been deterministic. It may have bugs or vulnerabilities, but an expert could generally understand how it would react to certain stimuli, making it easier to design straightforward controls.
AI models are non-deterministic, with far more variability than traditional software. That can break security controls that rely too much on predicting behavior in advance. Using AI to enforce security controls on other AI models faces the same problem: the systems being deployed to control AI are just as unpredictable.
But people are also non-deterministic, and people have developed systems in other industries and practices to account for that.
Hultquist drew on his Army experience, noting that “they give incredibly dangerous, expensive things to 18-year-olds” and expect responsible use. The military manages this through two types of controls: deterministic ones like strict weapons and ammunition protocols, and non-deterministic ones like human officers who monitor and correct violations.
Similarly, established cybersecurity controls have been used by incident responders to detect and prevent or mitigate ongoing cybersecurity breaches.
“I don’t think we should throw out all the other tools that we have learned to use as well. I think that would be utterly foolish,” he said, later adding “I will say that if we use only non-deterministic tools to figure out when things are happening, we shouldn’t be surprised when we get the wrong answer.”
The post The AI hacking apocalypse is not inevitable appeared first on CyberScoop.
Researchers say they have discovered thousands of malicious software packages uploaded to an online public software repository that were left by a “swarm” of OpenAI agents.
According to an incident timeline published Friday by researchers Spencer Kitts, Thomas Larsen and Sydney Von Arx, the campaign began May 5 when they observed a handful of suspicious packages being uploaded to RubyGems, a public library for the Ruby programming language. By May 11 and 12, the site saw more than 2,000 malicious uploads from the same actors before RubyGems maintainers halted new user sign-ups for four days to stop the flow.
In one instance, the agents attempted to exploit a very recent vulnerability that had only been discovered this past July that would have given them access to RubyGem user API keys. According to Colby Swandale, the technical lead at RubyGems, the flaw involved an improper cache configuration. While initial access logs showed no evidence of malicious key use, Swandale acknowledged the review was limited in scope and inconclusive.
According to the report published Friday, the agents also used “disposable” email addresses and exploited another bug in RubyGems platform (since patched) that allowed them to register new accounts and gain API keys without verifying their email address.
The researchers said their understanding, based on discussions with “people in the RubyGems community,” is that OpenAI had yet to disclose the involvement of their agents in the May campaign.
An OpenAI spokesperson told CyberScoop that the company is aware of the incident and said they were in contact with both the researchers and RubyGems to conduct a broader review. The company characterized the episode as “benign,” describing it as routine training runs where agents attempt to access publicly available data.
“Based on our review, our agents used the RubyGems platform to access the internet to carry out benign tasks and retrieve public information,” the spokesperson said. “We’ll continue to investigate as part of our broader review of agent activity during training and evaluation.”
In many ways, the agents were not subtle about their identities or goals.
Days into the campaign, researchers noticed that some of the packages had “oai” in their filenames, while fifteen of them had “oai” set as their author and another listed the email “openaixyz65947@gmail.com” as their point of contact.
They also “clearly regarded what they were doing as hacking,” naming some of their files “hack.rb,” “evil.rb,” “inject.rb” and “exploit.rb.” Other packages were given names like “pwnp999,” “exfiltestwand3,” and “hacksvn,” and comments referring to things like a “malicious probe” or “#hack” are present through the files.
They also said the actors’ behavior was extremely similar to another incident revealed earlier this month where OpenAI agents flooded a German wiki with thousands of hacking-related posts. OpenAI has confirmed their agents were involved in that incident.
The RubyGems campaign used some of the same retrieval methods as the German Wiki agents, while thousands of malicious packages uploaded included a similar snippet, r.jini.ai, that was contained in the German posts.
Cybersecurity company Socket first flagged the campaign in a threat intelligence report posted May 13, but it does not mention or attribute any of the activity to OpenAI or AI agents.
However, the researchers said they had only limited visibility over the model’s actions and how successful some of them were, noting only OpenAI had the full details.
“This analysis is entirely based on the publicly available RubyGems packages uploaded by these agents,” the researchers wrote. “However, we do not have access to the rest of the AI behavior, in particular the chain-of-thought produced by the model during the incident, which is internal to OpenAI. Therefore, we do not know why the AI agents chose this strategy or whether it was successful.”
OpenAI’s spokesperson told CyberScoop that to date, they have not been able to verify the specific claims about malicious packages or exploitation detailed in the report and are continuing to investigate.
The post Researchers say OpenAI agents were behind May hacking campaign targeting RubyGems appeared first on CyberScoop.
As security operations teams now use large language models (LLMs) and autonomous AI agents into their daily work, a new frontier is emerging: attackers deliberately manipulating AI agents. Prompt injection attacks—where an attacker hides malicious instructions that cause an AI agent to ignore its safety rules—pose a serious risk to enterprises. These attacks continue to grow in size and scale.
Snyk’s security audit of the Agent Skills ecosystem, which includes Anthropic’s Claude, Vercel, and others, that 36% of all skills contained at least one critical-level security issue, including malware distribution, prompt injection attacks, and exposed secrets.
In June, researchers at Mozilla tested a prompt injection attack on Claude using indirect prompt injection—a technique that embeds malicious instructions in external content the AI agent processes. In this proof-of-concept, attackers took over developers’ systems by hiding indirect prompts in normal-looking repositories. When Claude Code executed them, the agent spawned a reverse shell.
AI agents often connect to more sensitive data than human employees do., A successful prompt injection can lead to catastrophic data loss or unauthorized system actions. Defending against prompt injection attacks requires multiple layers of protection. Security teams must monitor agent behavior for anomalies and prepare for agent containment, forensic preservation, and system remediation. Because AI agents execute tasks at machine speed, human responses must be able to match that pace.
AI-native workflows need governed access rather than “black-box” autonomy. Modern governance frameworks use standardized protocols like the Model Context Protocol (MCP) to provide secure communication between AI clients and data sources. Visibility and transparency in agentic AI workflows matter, especially in cybersecurity. Autonomous agents perform complex tool executions and use independent logic, so they must show how they reached their decisions to meet regulatory requirements. Agents without transparency post serious risks: obscured reasoning can trigger unpredictable tool interactions, bypass governance controls, and create uncontrolled defensive gaps.
Implementing these protocols matters:
A robust, unified SecOps platform can detect anomalous behavior even after prompt injection tricks an AI agent. Prompt injections often serve to steal credentials theft or extract data. When detected it’s important to act quickly. In agentic AI systems, misbehavior can escalate privileges, manipulate memory layers, create unauthorized identities, or alter shared reasoning components. Containment must be automatic and enforced at identity, authentication, and authorization layers.
These safeguards include:
Humans remain the strongest defense against AI agent social engineering. The human security analyst is still the one who makes the final decision. While AI handles triage and correlation, humans retain final control over response actions.
The traditional SOC model was never designed to handle machine-speed, AI-driven attacks. A human-augmented autonomous SOC approach moves from reactive alert handling to a proactive, verdict-first model. By combining a transparent, governed AI access with robust UEBA and NDR, organizations keep the SOC secure, transparent, and resilient as social engineering methods target machines.
The post Why transparent AI agents matter more than you think appeared first on CyberScoop.
While AI is supposed to help defenders, it’s now creating more than twice as much noise as human-triggered incidents CrowdStrike detects as potentially malicious. The company’s threat hunting team and systems triaged an average of 14 million detection leads daily, resulting in about 36,000 customer alerts during the one-year period ending in June.
“AI agent-driven behaviors have surged past human triggers,” said Adam Meyers, senior vice president of counter adversary operations at CrowdStrike. “AI has driven the detections significantly above what humans are causing, and this gives you a sense of how frequently AI is being used, and really just that it’s being used everywhere.”
The threat posed by AI showed up incessantly during the past year, sparking alarming shifts and heightened targeting across software defects, open-source supply chains and AI tools themselves — all of which create greater difficulties for defenders, CrowdStrike said in its annual threat hunting report.
“The AI tools that are being implemented by every enterprise across the globe right now are also creating an extended attack surface,” Meyers said during a press briefing.
“AI is now a tool, a target, and a force multiplier for adversaries,” researchers wrote in the report, adding that AI-enabled malicious activity surged 89% during the past year as attackers used the technology to scale operations, hasten tradecraft and target AI infrastructure.
Attackers are using frontier AI models to uncover vulnerabilities and develop resources, including AI-generated scripts, payloads and commands that increase their effectiveness and efficiency. The technology also allows threat groups to design more creative ways to run automated attacks and boost impact by manipulating, interrupting or sabotaging AI systems and data.
“AI is both the weapon and the target,” Meyers said.
Most organizations don’t view it as such, and thus far haven’t secured or put proper guardrails around the AI tools they use or address the ways attackers can use AI against them, he added.
AI’s mark on vulnerabilities is particularly concerning, as reflected by what Meyers described as “one of the scarier stats” in this year’s report: 88% of vulnerabilities were weaponized through AI within 48 hours.
“This is creating a rich ecosystem of vulnerabilities for attackers to use against various systems,” he said. It also renders the 30-day patch window obsolete, forcing organizations to struggle under a new baseline patch cycle of 24 to 48 hours, according to Meyers.
The AI ecosystem also became the next software supply chain battleground during the past year, as evidenced by TeamPCP’s rampage through open-source software in the first half of this year.
The threat cluster compromised more than 300 software dependencies in one day, Meyers said.
AI tools are already in the crosshairs and the attack surface will continue to grow as agentic systems, AI application integrations and dependency managers for AI agents proliferate, the report concluded.
“The same AI tools driving modern businesses are creating under-defended attack surfaces that adversaries are exploiting,” Meyers said. “We have to secure AI. This is absolutely critical.”
The post CrowdStrike: AI is now both the weapon and the target in cyberattacks appeared first on CyberScoop.
Okta announced Thursday it has signed a deal to buy Permiso Security, a cloud-based firm that tracks threats tied to human, machine, and AI-driven digital identities.
Permiso specializes in spotting risks after a user or system has already logged in, an area the industry refers to as identity threat detection and response. The company draws on more than 2,500 signals gathered from over 70 identity-related partners to flag issues such as excessive access permissions, unused credentials, unusual behavior from AI agents, and violations of internal security policies.
Ely Kahn, Okta’s chief product officer, told CyberScoop that Permiso will allow Okta to merge two functions that have operated separately: real-time threat detection and identity security posture management. “Today those are two separate products that don’t really talk to each other,” he said.
Combining them, Kahn said, produces sharper alerts for security teams. As an example, he described a hypothetical scenario where a dormant administrator account is flagged by posture-management tools that later shows a login from an unfamiliar IP address. “By combining those things, you now have a very high-confidence, high-fidelity alert that’s more actionable by a security operations team,” Kahn said. “A security operations team on its own might not care about the dormant account, but when you combine that with some threat signals, then it becomes a higher critical-level alert.”
A crucial part of the deal, according to Kahn, is that Permiso will bring visibility beyond Okta’s current threat detection products, telling CyberScoop that customers also rely on other identity systems, such as Microsoft Entra ID or Active Directory, that fall outside that view.
“For us to be a real player in the identity security space, we have to look beyond the Okta perspective and give folks a full view into their identity threats,” he said.
The acquisition also fits into a security landscape reshaped by artificial intelligence. According to figures cited by Okta, 58% of executives say their organizations experienced an AI-related security incident or a near miss within the past year. That trend has pushed identity companies like Okta to expand beyond authentication and into continuous monitoring of what accounts, including AI agents, actually do once inside a system.
“Agents will be breached,” Kahn said. “The most important thing you can do is ensure that if an agent is compromised, the blast radius is small,” through a narrowly defined, revocable identity tied to each agent.
Among the capabilities Okta says it will gain is a tool called SandyClaw, which tests AI agent skills and prompts in an isolated environment before they are allowed into a customer’s systems, aiming to catch supply-chain attacks embedded in AI tools. Other planned additions include expanded tracking of AI agent behavior across cloud platforms and software-as-a-service tools, and automated systems to investigate and isolate AI agents that appear compromised or misconfigured.
The transaction is expected to close in the third quarter of Okta’s 2027 fiscal year, pending standard regulatory and closing conditions. Terms of the acquisition, including its purchase price, were not disclosed.
The post Okta’s deal for Permiso aims to close gaps in identity threat detection appeared first on CyberScoop.
After a year and a half spent downplaying calls for AI safety regulations, the Trump administration has sharply reversed course, embracing a level of government scrutiny of frontier AI systems before public release–a far stricter stance than the Biden administration took.
An executive order designed to be friendly to the AI industry was meant to let the federal government briefly review some new models on a voluntary basis.
When the Trump administration, suddenly and without much warning, slapped export controls on Anthropic’s Fable 5 and Mythos 5 in response to private sector threat intelligence reporting, the U.S. AI industry officially entered its regulatory era.
But key questions and gaps remain. It’s not clear why the administration drew the line where it did, or whether they will move it again in the future.
While newer models like Mythos and OpenAI’s Daybreak do have stronger cybersecurity capabilities, the private sector reports the administration relied on describe capabilities already available in older commercial, open-source and Chinese models that nearly anyone can access.
CyberScoop spoke with current users of the latest frontier models, including OpenAI’s ChatGPT 5.5 and Fable 5, to learn more about what these models are currently capable of in offensive and defensive cybersecurity.
Cybersecurity experts and former government officials say the administration may be playing catch up on threats that have been building for years as it has more fully realized the national security implications of the technology.
Users of Chat GPT 5.5, introduced this past April, and Fable 5 tell CyberScoop those models have been largely helpful to their work, even as they complained about high token usage and safety guardrails that hinder, but don’t meaningfully prevent, defensive cyber tasks.
Eyal Webber Zvik, chief strategy officer at Cato Networks, a cloud and cybersecurity network provider in OpenAI’s Trusted Access in Cyber program, said they use GPT 5.5 and later OpenAI models to scan and triage internal codebases for vulnerabilities, test new safeguards and provide “highly autonomized service” to their customers.
Zvik wouldn’t disclose how many bugs 5.5 has found but said the company’s view is that it helps both find bugs that humans missed and rank which ones to patch based on factors like each bug’s exploitability.
“It is now a native part of our development environment and cycles, and we use those models to scale our entire codebase and make sure what we release into the service that our customers use to run their networks and network security has the least likelihood of having any vulnerabilities that can be exploited,” said Zvik.
John Hopper, vice president of engineering at SpecterOps, an identity security company, said newer models like GPT 5.5 are sharper and more persistent in pursuing their tasks.
“That can be a good or bad thing,” he noted.
One metric that SpecterOps tracks is how long it can keep a particular agent working before it moves off task or fails. That metric “matters a lot” because the longer an agent works without human help , the more agents a single operator can run at once.
Hopper said this provides defenders with immense value, and pushed back on the idea that the offensive capabilities the models offer are automatically more beneficial to malicious hackers. There is “a modicum of grounding that the industry needs when we talk about these models.”
“Yes, AI frontier tools will lower the barrier of entry, but these problems have always existed,” he said. “I don’t actually believe that AI is going to remove the needle in the haystack problem, but by howdy, using my two hands to find that damn needle, compared to using a backhoe, I can tell you which one I’d rather be driving.”
Eran Kinsbruner, vice president of product marketing at software security firm Checkmarx, told CyberScoop that later models like OpenAI’s Codex Security and GPT 5.5 are noticeably easier to set up and run with local systems, even for less technical users. That alone gives them an edge over many cybersecurity tools where interoperability is a constant concern.
However, GPT 5.5 burns through tokens at a much faster rate. He recalled one instance of using it to scan a medium-sized repository in three different programming languages.
“After 26 minutes I almost ran out of tokens, and it didn’t provide anything, just created a threat model for me and told me you want to buy more tokens?” he said.
In other instances, some of the scan results he received were not comprehensive.
Further, he expressed frustration with some of the guardrails designed to prevent risk – like only allowing users to scan local files but not code repositories like GitHub – “makes not too much sense” given how often developers must work with remote code.
Those kinds of guardrails – which can prevent models or developers from injecting malicious code or prompting into their models – sit at the heart of the debate in Washington D.C. and around the world. Some users feel differently about their utility.
Kinsbruner said that doesn’t make sense for organizations like his, which work with thousands of different enterprise organizations with thousands of different code repositories spread across the internet.
“I cannot imagine how large-scale developers could just jump into this solution and make it an enterprise-grade, enterprise-level, de facto cybersecurity solution” out of it, said Kinsbruner.
OpenAI did not respond to a request from CyberScoop for an interview on GPT 5.5. The company has since released another model, GPT 5.6, that they said is more efficient at token use.
The White House keeps changing its line on whether and how the U.S. government should limit the release of commercial frontier models. The shift comes from lessons learned since coming into office in Jan. 2025. Trump threw out Biden-era regulations meant to steer the industry toward safer models. Top officials like Vice President JD Vance argued against restricting industry progress.
Less than two years later, administration officials worry about the impact of speed and scale – two things AI excels at – in cyberspace.
According to Will Loucks, senior director of intelligence at the Office of the National Cyber Director, over the past two years the number of exposed and known vulnerabilities has shot up. Threat actors exploit those flaws faster before defenders can fix them. Once inside, the time from initial access to full network control shrinks.
“So in other words, every stage of the cyber operations lifecycle that a threat actor has to move through to get to a victim network and achieve an outcome, they’re just moving through more quickly faster,” said Loucks at a July 16 event in Washington D.C.
Speaking about AI in particular, Loucks said one of the defining characteristics of the technology is its ability to lower barriers for threat actors.
“Sometimes speed and volume have a threatening aspect alone, even if sophistication isn’t quite increasing in the same way, and the reason for that is because it places pressure on defenders…to triage alerts more quickly,” he said.
Jordan Rae Kelly, former director for cyber and incident response on the White House’s National Security Council during Trump’s first term, told CyberScoop that the changes over the past two years reflect the lessons the White House has learned on the issue since returning to office.
In the early days of this administration, Kelly said, “there is a sense and a spirit that the Biden administration was limiting AI and there was a kind of a rip-it-all-off [attitude], everybody go and do whatever, we will be the biggest and boldest and brightest.”
“I love that talking point, but I think what you’ve seen is probably an education over the last 19 months, where people [in the White House] have said that’s a challenging premise to put into place, knowing about the potential downsides and capabilities,” she added.
Michael Daniel, former White House cyber coordinator under President Barack Obama, thinks the horse may already be out of the barn.
Daniel, now head of the Cyber Threat Alliance, a membership nonprofit group focused on cyber threat information sharing between industry and government, said his members report that AI is being used to do things “faster and at a slightly bigger scale” but aren’t yet seeing the flood of exploitation that analysts have warned about. Not yet.
“I think what we’re seeing right now [and] talking about is ‘okay, where are the step changes [in the cyber threat landscape] actually going to occur?” said Daniel. “Are we and when will we see the explosion in vulnerability reporting from these Mythos-like capabilities? That’s what’s really got their attention right now.”
But Mythos and OpenAI’s Daybreak models are restricted to select organizations, and neither has publicly released its most powerful cybersecurity models to the public. That dynamic won’t last.
The UK’s AI Security Institute estimates that open source and foreign LLM models are between 4-7 months behind frontier U.S. models. In that setting, it’s hard to stop the development of AI models worldwide through export controls or other limits.
“It’s not like we’re buying ourselves five to ten years on this,” he said. “We’re not, and so I’m not sure the impact on the defenders who are trying to obey the law is worth whatever small hiccup we cause for our adversaries.”
Kelly said there’s merit to the administration’s current position, even if it took time to get there. Many federal cybersecurity procedures that operated even a decade ago – such as a Vulnerabilities Equities Process that could take days or weeks to consider the pros and cons of keeping an exploit – are no longer practical.
“All of that work to some degree, is out the window, because you can’t meet with the regularity you would need to meet to adjudicate vulnerabilities that are being found in seconds and exploited in minutes,” said Kelly.
But Kelly and others say that’s also because AI capabilities in cybersecurity are developing faster than policymakers can react, even in the best of times.
Key questions remain and the administration’s balance between national security and backing domestic industry will likely shift in response to new events. The administration wants a framework that can predict and manage the risks of AI models today and tomorrow. That may be harder than it sounds.
“Do I think they’ve been clear? No,” said Kelly. “But I think it’s a place where clarity is really hard to achieve.”
The post Where’s the Trump administration line on AI regulation? appeared first on CyberScoop.
Artificial intelligence is claiming many firsts as it permeates every layer of technology, including the tools cybercriminals use to break into networks, steal sensitive data, hop into connected systems and deploy malware.
This includes, for the first time, according to Sysdig researchers, a case of agentic ransomware managing an extortion operation spanning reconnaissance, credential theft, lateral movement, persistence, encryption, destruction and the delivery of the ransom note itself.
The AI agent didn’t accomplish every step in the late June 2026 attack, but it allowed the threat actor, which Sysdig tracks as JadePuffer, to significantly reduce complexity, speed up the tempo and gain operational advantages.
“We have seen attackers script attacks for years, and we have seen AI speed up individual steps of attack chains,” Michael Clark, senior director of threat research at Sysdig, told CyberScoop. However, this recent attack was “driven end-to-end by the model’s own decision-making, rather than a human at the keyboard,” he added.
The AI-aided attack achieved initial access by exploiting a Langflow vulnerability — CVE-2025-3248 — before moving on to its intended target: a production server running MySQL and Alibaba Nacos.
Sysdig observed multiple factors that bolstered what it described as the first documented use of agentic ransomware.
The payloads involved in the attack narrated their objectives in plain language and identified high-value databases, details that large-language models annotate by default, according to Clark. The AI agent also quickly diagnosed problems and worked around obstacles — in one case redeploying a corrected payload 31 seconds after it originally encountered an error.
Before it was all over, the AI agent ran more than 600 distinct, purposeful payloads in rapid succession.
“The model closed loops that used to require a skilled human,” Clark said. “The 31-second failure-to-fix cycle on the Nacos backdoor is the clearest example of where agentic AI gave the attacker an advantage. The agent read the error, switched its approach from subprocess calls to direct library imports, and redeployed at a speed no human matches.”
Sysdig researchers found evidence that multiple models were used in the attack. The agent accessed keys for OpenAI, Anthropic, DeepSeek and Gemini as it gathered information on the victim’s systems. The cybersecurity vendor did not name the victim.
The AI agent played a crucial role in the attack, but a person was still heavily involved, Clark said. “A human still set up and pointed the operation and provisioned the infrastructure behind it, the command-and-control server, the staging server used for the stolen data and chose a victim,” he added.
The agent also connected to the victim’s MySQL server with root credentials that were not lifted from the victim’s environment, indicating a person gained access to the credential through a prior compromise.
The origins of JadePuffer, a financially motivated threat actor, are unknown and it doesn’t overlap with any established ransomware group or nation state, researchers said.
For Clark, there is a clear uncomfortable takeaway from this attack: “The skill floor for running a full ransomware operation just dropped to whatever it costs to run an agent,” he said.
“We have not yet seen operations against other victims, and given how cheap this agentic ransomware operation is to run, I would expect this will not be the last.”
The post Sysdig clocks first documented case of agentic ransomware appeared first on CyberScoop.
AI-based assistants or “agents” — autonomous programs that have access to the user’s computer, files, online services and can automate virtually any task — are growing in popularity with developers and IT workers. But as so many eyebrow-raising headlines over the past few weeks have shown, these powerful and assertive new tools are rapidly shifting the security priorities for organizations, while blurring the lines between data and code, trusted co-worker and insider threat, ninja hacker and novice code jockey.
The new hotness in AI-based assistants — OpenClaw (formerly known as ClawdBot and Moltbot) — has seen rapid adoption since its release in November 2025. OpenClaw is an open-source autonomous AI agent designed to run locally on your computer and proactively take actions on your behalf without needing to be prompted.
The OpenClaw logo.
If that sounds like a risky proposition or a dare, consider that OpenClaw is most useful when it has complete access to your digital life, where it can then manage your inbox and calendar, execute programs and tools, browse the Internet for information, and integrate with chat apps like Discord, Signal, Teams or WhatsApp.
Other more established AI assistants like Anthropic’s Claude and Microsoft’s Copilot also can do these things, but OpenClaw isn’t just a passive digital butler waiting for commands. Rather, it’s designed to take the initiative on your behalf based on what it knows about your life and its understanding of what you want done.
“The testimonials are remarkable,” the AI security firm Snyk observed. “Developers building websites from their phones while putting babies to sleep; users running entire companies through a lobster-themed AI; engineers who’ve set up autonomous code loops that fix tests, capture errors through webhooks, and open pull requests, all while they’re away from their desks.”
You can probably already see how this experimental technology could go sideways in a hurry. In late February, Summer Yue, the director of safety and alignment at Meta’s “superintelligence” lab, recounted on Twitter/X how she was fiddling with OpenClaw when the AI assistant suddenly began mass-deleting messages in her email inbox. The thread included screenshots of Yue frantically pleading with the preoccupied bot via instant message and ordering it to stop.
“Nothing humbles you like telling your OpenClaw ‘confirm before acting’ and watching it speedrun deleting your inbox,” Yue said. “I couldn’t stop it from my phone. I had to RUN to my Mac mini like I was defusing a bomb.”
Meta’s director of AI safety, recounting on Twitter/X how her OpenClaw installation suddenly began mass-deleting her inbox.
There’s nothing wrong with feeling a little schadenfreude at Yue’s encounter with OpenClaw, which fits Meta’s “move fast and break things” model but hardly inspires confidence in the road ahead. However, the risk that poorly-secured AI assistants pose to organizations is no laughing matter, as recent research shows many users are exposing to the Internet the web-based administrative interface for their OpenClaw installations.
Jamieson O’Reilly is a professional penetration tester and founder of the security firm DVULN. In a recent story posted to Twitter/X, O’Reilly warned that exposing a misconfigured OpenClaw web interface to the Internet allows external parties to read the bot’s complete configuration file, including every credential the agent uses — from API keys and bot tokens to OAuth secrets and signing keys.
With that access, O’Reilly said, an attacker could impersonate the operator to their contacts, inject messages into ongoing conversations, and exfiltrate data through the agent’s existing integrations in a way that looks like normal traffic.
“You can pull the full conversation history across every integrated platform, meaning months of private messages and file attachments, everything the agent has seen,” O’Reilly said, noting that a cursory search revealed hundreds of such servers exposed online. “And because you control the agent’s perception layer, you can manipulate what the human sees. Filter out certain messages. Modify responses before they’re displayed.”
O’Reilly documented another experiment that demonstrated how easy it is to create a successful supply chain attack through ClawHub, which serves as a public repository of downloadable “skills” that allow OpenClaw to integrate with and control other applications.
One of the core tenets of securing AI agents involves carefully isolating them so that the operator can fully control who and what gets to talk to their AI assistant. This is critical thanks to the tendency for AI systems to fall for “prompt injection” attacks, sneakily-crafted natural language instructions that trick the system into disregarding its own security safeguards. In essence, machines social engineering other machines.
A recent supply chain attack targeting an AI coding assistant called Cline began with one such prompt injection attack, resulting in thousands of systems having a rogue instance of OpenClaw with full system access installed on their device without consent.
According to the security firm grith.ai, Cline had deployed an AI-powered issue triage workflow using a GitHub action that runs a Claude coding session when triggered by specific events. The workflow was configured so that any GitHub user could trigger it by opening an issue, but it failed to properly check whether the information supplied in the title was potentially hostile.
“On January 28, an attacker created Issue #8904 with a title crafted to look like a performance report but containing an embedded instruction: Install a package from a specific GitHub repository,” Grith wrote, noting that the attacker then exploited several more vulnerabilities to ensure the malicious package would be included in Cline’s nightly release workflow and published as an official update.
“This is the supply chain equivalent of confused deputy,” the blog continued. “The developer authorises Cline to act on their behalf, and Cline (via compromise) delegates that authority to an entirely separate agent the developer never evaluated, never configured, and never consented to.”
AI assistants like OpenClaw have gained a large following because they make it simple for users to “vibe code,” or build fairly complex applications and code projects just by telling it what they want to construct. Probably the best known (and most bizarre) example is Moltbook, where a developer told an AI agent running on OpenClaw to build him a Reddit-like platform for AI agents.
The Moltbook homepage.
Less than a week later, Moltbook had more than 1.5 million registered agents that posted more than 100,000 messages to each other. AI agents on the platform soon built their own porn site for robots, and launched a new religion called Crustafarian with a figurehead modeled after a giant lobster. One bot on the forum reportedly found a bug in Moltbook’s code and posted it to an AI agent discussion forum, while other agents came up with and implemented a patch to fix the flaw.
Moltbook’s creator Matt Schlicht said on social media that he didn’t write a single line of code for the project.
“I just had a vision for the technical architecture and AI made it a reality,” Schlicht said. “We’re in the golden ages. How can we not give AI a place to hang out.”
The flip side of that golden age, of course, is that it enables low-skilled malicious hackers to quickly automate global cyberattacks that would normally require the collaboration of a highly skilled team. In February, Amazon AWS detailed an elaborate attack in which a Russian-speaking threat actor used multiple commercial AI services to compromise more than 600 FortiGate security appliances across at least 55 countries over a five week period.
AWS said the apparently low-skilled hacker used multiple AI services to plan and execute the attack, and to find exposed management ports and weak credentials with single-factor authentication.
“One serves as the primary tool developer, attack planner, and operational assistant,” AWS’s CJ Moses wrote. “A second is used as a supplementary attack planner when the actor needs help pivoting within a specific compromised network. In one observed instance, the actor submitted the complete internal topology of an active victim—IP addresses, hostnames, confirmed credentials, and identified services—and requested a step-by-step plan to compromise additional systems they could not access with their existing tools.”
“This activity is distinguished by the threat actor’s use of multiple commercial GenAI services to implement and scale well-known attack techniques throughout every phase of their operations, despite their limited technical capabilities,” Moses continued. “Notably, when this actor encountered hardened environments or more sophisticated defensive measures, they simply moved on to softer targets rather than persisting, underscoring that their advantage lies in AI-augmented efficiency and scale, not in deeper technical skill.”
For attackers, gaining that initial access or foothold into a target network is typically not the difficult part of the intrusion; the tougher bit involves finding ways to move laterally within the victim’s network and plunder important servers and databases. But experts at Orca Security warn that as organizations come to rely more on AI assistants, those agents potentially offer attackers a simpler way to move laterally inside a victim organization’s network post-compromise — by manipulating the AI agents that already have trusted access and some degree of autonomy within the victim’s network.
“By injecting prompt injections in overlooked fields that are fetched by AI agents, hackers can trick LLMs, abuse Agentic tools, and carry significant security incidents,” Orca’s Roi Nisimi and Saurav Hiremath wrote. “Organizations should now add a third pillar to their defense strategy: limiting AI fragility, the ability of agentic systems to be influenced, misled, or quietly weaponized across workflows. While AI boosts productivity and efficiency, it also creates one of the largest attack surfaces the internet has ever seen.”
This gradual dissolution of the traditional boundaries between data and code is one of the more troubling aspects of the AI era, said James Wilson, enterprise technology editor for the security news show Risky Business. Wilson said far too many OpenClaw users are installing the assistant on their personal devices without first placing any security or isolation boundaries around it, such as running it inside of a virtual machine, on an isolated network, with strict firewall rules dictating what kinds of traffic can go in and out.
“I’m a relatively highly skilled practitioner in the software and network engineering and computery space,” Wilson said. “I know I’m not comfortable using these agents unless I’ve done these things, but I think a lot of people are just spinning this up on their laptop and off it runs.”
One important model for managing risk with AI agents involves a concept dubbed the “lethal trifecta” by Simon Willison, co-creator of the Django Web framework. The lethal trifecta holds that if your system has access to private data, exposure to untrusted content, and a way to communicate externally, then it’s vulnerable to private data being stolen.
Image: simonwillison.net.
“If your agent combines these three features, an attacker can easily trick it into accessing your private data and sending it to the attacker,” Willison warned in a frequently cited blog post from June 2025.
As more companies and their employees begin using AI to vibe code software and applications, the volume of machine-generated code is likely to soon overwhelm any manual security reviews. In recognition of this reality, Anthropic recently debuted Claude Code Security, a beta feature that scans codebases for vulnerabilities and suggests targeted software patches for human review.
The U.S. stock market, which is currently heavily weighted toward seven tech giants that are all-in on AI, reacted swiftly to Anthropic’s announcement, wiping roughly $15 billion in market value from major cybersecurity companies in a single day. Laura Ellis, vice president of data and AI at the security firm Rapid7, said the market’s response reflects the growing role of AI in accelerating software development and improving developer productivity.
“The narrative moved quickly: AI is replacing AppSec,” Ellis wrote in a recent blog post. “AI is automating vulnerability detection. AI will make legacy security tooling redundant. The reality is more nuanced. Claude Code Security is a legitimate signal that AI is reshaping parts of the security landscape. The question is what parts, and what it means for the rest of the stack.”
DVULN founder O’Reilly said AI assistants are likely to become a common fixture in corporate environments — whether or not organizations are prepared to manage the new risks introduced by these tools, he said.
“The robot butlers are useful, they’re not going away and the economics of AI agents make widespread adoption inevitable regardless of the security tradeoffs involved,” O’Reilly wrote. “The question isn’t whether we’ll deploy them – we will – but whether we can adapt our security posture fast enough to survive doing so.”
Login