Hacking with AI: How to prevent your AI from going rogue

By Ayoub

November 19, 2026

Hacking with AI: How to prevent your AI from going rogue Header Image

AI has reshaped many industries, including bug bounty. More and more security researchers are deploying AI to perform all sorts of web hacking tasks. From automating reconnaissance to identifying complex vulnerability chains, AI models have become a powerful addition to any researcher’s toolkit.

However, as with many things, when used improperly, these models can cause serious damage through intrusive testing. An AI model without sufficient guardrails lacks context around program scope, doesn’t respect program-specific rules, and may also not understand when to stop and prompt for human intervention.

In this article, we’ll explore why unguarded AI models pose a real risk during security testing, why researchers bear full responsibility for their models’ actions, and what guidelines you should follow before deploying an AI model against any target.

Let’s dive in!

The problem with unguarded AI models

A bug bounty program or responsible disclosure policy is an agreement between the security researcher and the company. The company behind the policy defines the scope, sets requirements, and, in return, provides you with the authorization to test its assets. As long as the researcher operates within those boundaries, their testing is authorized.

However, the moment testing moves outside the defined scope, whether by actively testing out-of-scope assets or resorting to intrusive techniques (such as exceeding rate limits or testing for DDoS), that authorization no longer applies. This is worth keeping in mind as we discuss AI-assisted testing, because an AI model doesn’t understand this agreement until you provide it the context it lacks.

When given an objective like “find vulnerabilities on target X,” an AI model will pursue that objective as aggressively as its configuration allows. But that doesn't necessarily mean it'll distinguish an in-scope from an out-of-scope asset, nor will it self-limit its request volume, or ever consider whether its behavior resembles a real attack.

In practice, this means that an unguarded AI model can trigger thousands of requests against production endpoints in minutes, interact with assets the program never authorized (such as signing up for accounts in out-of-scope assets), and generate alerts that are not immediately distinguishable from a genuine intrusion. For the target company’s security team, such incidents are often treated as a real incident until proven otherwise, and that response comes at a real cost.

Owning responsibility for AI models

It is essential to note that an AI model is a tool, no different from a scanner or a custom script. When a researcher deploys a model that crosses scope boundaries or violates program rules, the responsibility lies with the researcher, not the model. “The AI did it” is not a valid defense, just as “my scanner did it” has never been one.

As researchers, we are responsible for every action our tools take on our behalf. This applies to AI models just as much as it applies to anything else in our toolkit. If you deploy it, you are responsible for any of its actions.

Guidelines for responsible AI deployment

Now that we understand the risks and where responsibility lies, let’s have a look at a set of guidelines every researcher should consider before deploying an AI model against any target.

Defining & enforcing program engagement rules

Before deploying an AI model, the program’s engagement rules should be part of your primary context window or initial KB of your model’s configuration. Sadly, even though most models are quite advanced, simple prompt-level instructions alone are not reliable. Telling a model 'only test *.example.com' in a system prompt does not guarantee it will always comply.

Additionally, engagement rules extend beyond just the scope. Programs often define specific requirements for how you should conduct your testing. For instance, you may be asked to use your Intigriti email alias when creating test accounts, to include a specific request header, or to avoid intrusive testing such as testing for DoS vulnerabilities. These are requirements your AI model needs to follow just as strictly as the program's in-scope assets.

You should, therefore, strongly consider implementing a proxy with rules that define a list of allowed in-scope assets so the model has no ability to reach anything outside the defined scope. If the model can’t reach it, it can’t test it.

Restrict allowed actions and tools

To further gain control over your model, you will need to restrict its capabilities by defining the set of tools it is allowed to use. For instance, you can configure your Model with an MCP that allows it to browse web pages in a browser, provide it with access to CLI tools, but restrict any fuzzing tools that could result in intrusive testing.

Additionally, it's also recommended that you prompt the model to ask for human intervention when necessary. This is especially important when configuring your model to engage in active exploitation and not only identifying bugs. Allowing your model to probe for SQL injection vulnerabilities is easy, but having it actually exploit and retrieve or even permanently delete data within a database often goes against program engagement rules and will most likely trigger internal alerts.

Importance of logging

Keeping logs is not optional. Each HTTP request, TCP connection, action, and decision the model takes must be logged and kept for a certain amount of time. These logs will serve two main purposes.

First, if something goes wrong and a target’s security team reaches out, you need to be able to show exactly what the model did, when, and why. This is your evidence that the incident was unintentional and that you acted in good faith.

Secondly, logs help you identify where the model exactly deviated from its intended behavior, which goes without saying is essential for improving your setup.

Testing in controlled environments

Before pointing your model at a live target, run it against a controlled environment first. And ensure you've set it up with the scope configuration and restrictions you intend to run it against your target. This ensures two things: 1) that any scope rules are correctly enforced and 2) that your model follows any of your pre-defined testing engagement and restrictions when it's actively testing.

Things to question are: does it only test on in-scope assets, does it respect the rate limit, is it using your Intigriti email alias, will it add the mandatory request header to each request, will it go forward with possible intrusive testing or will it prompt for human intervention, etc.

It only takes a small amount of extra effort, but it can easily help prevent the kind of incidents we’ve discussed throughout this article.

Conclusion

AI-assisted security testing is here to stay, and for good reason. It allows researchers to scale their efforts and uncover vulnerabilities that might otherwise go unnoticed. However, deploying an AI model without sufficient guardrails puts targets, programs, and the research at risk. In this article, we’ve explored why unguarded AI models are a growing problem, why researchers bear full responsibility for their models’ actions, and what guidelines to follow before deploying a model against any target.

So, you've just learned something new about exploiting LLM prompt injection vulnerabilities... Right now, it's time to put your skills to the test! You can start by practicing on vulnerable labs and CTFs or... browse through our 70+ public bug bounty programs on Intigriti, and who knows, maybe earn a bounty on your next submission!

START HACKING ON INTIGRITI TODAY

Author

Ayoub

Senior security content developer, CVE owner, bug bounty hunter & former web app pentester

You may also like

Large language models (LLMs) are quickly becoming a standard component of any modern web application nowadays, powering everything from customer support chatbots to internal knowledge assistants. For us, security researchers, this opens a whole new attack surface to explore. In this article, we'll f

Read more: Exploiting LLM prompt injection vulnerabilities

As AI agents are deployed to automate more tasks, they become more capable. And as the famous quote goes: "With great power comes great responsibility." Assuming that humans in the loop can mitigate that risk turns out to be. At Bug Bounty Village during DEF CON 34, Inti De Ceukelaire, Founding Memb

Read more: Hacking AI customer service agents

Fuzzing has been around for as long as web applications have. In fact, the term itself was coined back in 1988, when Barton Miller, a professor at the University of Wisconsin, was working over a dial-up connection during a thunderstorm and noticed that the resulting line noise was consistently crash

Read more: Web fuzzing for hackers