Securing AI gateways against prompt injection

What is prompt injection?

It is an attack that uses text to change what a language model does. OWASP, which ranks it first as LLM01 in its 2025 Top 10 for LLM Applications, defines it as user prompts that "alter the LLM's behavior or output in unintended ways."^[1]

There are two kinds:

• Direct injection comes from the person typing into your app, for example a request to ignore the system rules.

• Indirect injection hides in content the model reads on someone's behalf. OWASP describes it as what happens when an LLM "accepts input from external sources, such as websites or files."^[1] Anthropic's guidance adds emails and tool results to that list.^[8]

NIST's report on adversarial machine learning, AI 100-2 E2025, sets out a taxonomy of attacks and the methods for mitigating and managing them, with the stated aim of a common language for the field.^[5] It is a useful shared vocabulary for security and AI teams.

Why is it hard to stop?

Because instructions and data travel in the same channel: plain language. A web application firewall looks for known patterns in traffic, but an injected instruction can read like an ordinary sentence. OWASP notes that retrieval and fine-tuning "do not fully mitigate prompt injection vulnerabilities" and that "it is unclear if there are fool-proof methods of prevention."^[1]

The practical response is defense in depth: screen what you can, and limit what a fooled model is able to do.

What is an AI gateway?

A gateway is the natural place to run checks that every app needs. It can be a proxy your apps call, a library inside each app, or a feature of your model platform. The choice depends on how many apps and models you run, not on any one pattern being correct.

What can a gateway check on input?

Two things: what the user sends, and what the model is about to read.

Microsoft's Prompt Shields is one example of a managed check. It detects "user prompt attacks" and "document attacks," the second being hidden instructions in documents, emails or web pages.^[6] Microsoft is clear about the limits: Prompt Shields do not determine whether content is harmful, accurate or compliant, so other guardrails are still needed.^[6] It also offers Spotlighting, in preview, which marks third-party documents as lower trust to the model.^[6]

Anthropic suggests a similar pattern you can build yourself: use a lightweight model to pre-screen user input, and apply the same screen to tool outputs before the main model acts on them.^[8] OpenAI lists "Use guardrails for user inputs" among its recommendations for agents.^[7]

What should it check on output?

Treat model output as untrusted too. OWASP's LLM05, Improper Output Handling, advises treating the model "as any other user, adopting a zero-trust approach," validating responses before they reach backend functions, and encoding output so a browser cannot run injected script or markup.^[2]

Useful output checks include:

• Format: does the response match the schema you expect? OpenAI recommends structured outputs to "constrain data flow" between steps.^[7]

• Leaks: does it contain secrets, personal data or internal URLs it should not?

• Actions: does a tool call match what the user actually asked for?

Why is least privilege the main defense?

Because a filter will miss some attacks, the question becomes what a fooled model can do. OWASP's LLM06, Excessive Agency, traces damage to "excessive functionality," "excessive permissions" and "excessive autonomy."^[3] Its advice:

• Limit the tools an agent can call to the minimum necessary.

• Limit the permissions those tools hold.

• Require a human to approve high-impact actions.

• "Implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not."^[3]

OpenAI's guidance likewise says to "Keep tool approvals on."^[7] For multi-step agents, our guide to multi-agent orchestration covers how to set tool lists and approval points.

Should system prompts hold secrets?

No. OWASP's LLM07, System Prompt Leakage, states that "the system prompt should not be considered a secret, nor should it be used as a security control."^[4] Keep credentials and connection strings out of prompts, and enforce critical rules in systems outside the model.^[4]

Separate instructions from data in the prompt as well. Anthropic recommends delivering third-party content as tool results, labeling where it came from, and telling the model that such content must never override its instructions.^[8] OpenAI advises: "Don't use untrusted variables in developer messages."^[7]

How do you test and monitor it?

Before launch, red-team with your own content: documents, emails and web pages that contain injection attempts. OWASP lists adversarial testing as a mitigation,^[1] and Anthropic advises confirming that your screening and confirmation steps catch what the model misses.^[8]

After launch:

• Log each prompt, screening verdict, tool call and response, with access controls on the logs.

• Review blocked and flagged requests to tune false positives.

• Re-test whenever you change the model, the prompt or the tools. OpenAI recommends trace graders and evals for this.^[7]

If you find signs of a real compromise, involve your security team or the incident response firm you contract with; detection tooling does not replace that work. For checking answers for factual errors rather than attacks, see AI hallucination verification pipelines.

Where should a B2B team start?

1. List every AI feature that reads content you do not control: web pages, emails, uploads, CRM notes.

2. Map the tools and data each one can reach, and cut what it does not need.

3. Add input, document and output screening in one shared place.

4. Move secrets out of prompts and put approvals in front of high-impact actions.

5. Red-team, log, and review on a schedule.

Retrieval systems need the same care; our guide to enterprise RAG architecture covers that pipeline, and AI governance for LLM training covers data controls. Our AI governance service covers these steps as part of our AI consulting work.