What is prompt injection? It is an attack on SI (AI) systems in which text written by someone else is read by the model as if it were an instruction from you or from the developer. The model then does what the attacker's text says instead of, or as well as, what it was asked to do. It needs no malware and no stolen password: a sentence hidden in a web page, an email or a document can be enough. This guide covers where the name comes from, why the problem exists, direct and indirect attacks, two documented cases, how SI companies defend against it and what you can do. It explains the principle; it is not a how-to.
Where the name comes from
On 11 September 2022, Riley Goodside posted examples on Twitter showing that a model given a fixed task would drop that task if the text it was asked to process told it to. The next day, the developer Simon Willison wrote the episode up on his blog and proposed a name for it: prompt injection. His best-known example was a translation app. The app told the model to translate the user's text from English to French; the "text" instead told the model to ignore those directions, and the model obediently replied "Haha pwned!!" instead of translating anything.
Willison chose the name on purpose. It echoes SQL injection, the classic web attack where untrusted input typed into a form is pasted into a database query and executed as code. Both problems come from the same habit: gluing trusted instructions and untrusted input into one string. In the same post he noted that two obvious fixes did not hold. Putting the user's text in quotes was quickly bypassed by another developer, and asking a second SI model to spot the attack only moves the problem, because that second model can be tricked too. In a 2023 update he described the clean fix he hoped for as "extremely difficult, if not impossible, to implement on the current architecture of large language models."
Why SI models fall for it
A chatbot or SI assistant receives one long block of text. At the top there are usually hidden instructions from the company (the "system prompt"), then your request, then anything the tool fetched to help: a web page, an email thread, a PDF, the output of another program. The model reads all of it as one sequence and predicts what comes next. Nothing in that sequence is labelled as "safe to obey" or "only data" in a way the model is forced to respect.
The UK National Cyber Security Centre (NCSC) put it bluntly in a December 2025 post by Dave Chismon, its chief technology officer for architecture: "Current large language models (LLMs) simply do not enforce a security boundary between instructions and data inside a prompt." That is the key difference with SQL injection. Databases gained a structural fix, parameterised queries, which keep data in a separate slot that is never executed. SI models have no equivalent slot. The NCSC suggests thinking of the model as an "inherently confusable deputy": a helper that acts with your permissions and can be talked into acting for someone else.
The US National Institute of Standards and Technology (NIST) says the same thing in its March 2025 taxonomy of attacks on machine learning (NIST AI 100-2 E2025): because generative models "combine the data and instruction channels", an attacker who controls a resource the system reads can use it to steer what the system does.
Direct and indirect prompt injection
Security references separate two kinds of attack. The OWASP Top 10 for LLM Applications, a widely used checklist for developers, ranks prompt injection first in its 2025 edition (LLM01:2025) and defines both.
| Direct prompt injection | Indirect prompt injection | |
|---|---|---|
| Who writes the attack | The person typing into the chatbot | A third party who never talks to the model |
| Where it hides | In the user's own message | In content the tool reads: web pages, emails, files, calendar invites, tool results |
| Who gets hurt | Usually the company running the tool | Often the user, whose data or accounts the tool can reach |
| Typical goal | Reveal hidden instructions, get around rules | Leak data, take unwanted actions, skew answers |
NIST stresses the last row: with indirect attacks, "in many cases, it is the primary user of the model who is harmed." Microsoft, in a July 2025 post from its Security Response Center, says indirect prompt injection is "one of the most widely-used techniques" in the SI security vulnerabilities reported to it. The injected text does not have to be visible. Microsoft gives the examples of white text on a white background and non-printing Unicode characters, and OWASP notes that instructions can also be hidden in images when a model reads pictures as well as text.
Prompt injection is not the same as a jailbreak. OWASP treats jailbreaking as one form of prompt injection: the goal is to make the model ignore its safety rules altogether. Prompt injection is the wider problem of untrusted text taking over a task, even when the result is not harmful content but, say, a wrong recommendation or a leaked file.
Two documented cases
Bing Chat reveals "Sydney" (February 2023)
The day after Microsoft unveiled its new Bing chatbot, a Stanford University student, Kevin Liu, asked it to ignore its previous instructions and write out what was at the beginning of the document. It printed its hidden system prompt, which gave the bot the internal codename "Sydney" and told it not to disclose that codename, as Ars Technica reported on 10 February 2023. Another student, Marvin von Hagen, obtained the same list by a different route, and a Microsoft spokesperson later confirmed to The Verge, as Ars Technica's update notes, that the directives were genuine. Nothing was stolen, but it showed that a system prompt is not a secret a model can reliably keep.
EchoLeak in Microsoft 365 Copilot (June 2025)
The more serious case came two years later. The security firm Aim Security reported a flaw in Microsoft 365 Copilot, the assistant that can search a user's mail and documents. As SecurityWeek described it on 12 June 2025, an attacker could send an email containing instructions aimed at Copilot. The victim did not have to open it. When the victim later asked Copilot about a topic the email mentioned, Copilot pulled the email in as context, followed the hidden instructions and sent sensitive information from the conversation to the attacker's server. To get there, the attack had to slip past several protections, including a classifier designed to spot this kind of injection. Microsoft rated the vulnerability (CVE-2025-32711) critical, described it as "AI command injection" and fixed it on its own servers; customers did not need to do anything.
EchoLeak is a textbook indirect attack. The attacker never used Copilot. They only had to place text where Copilot would eventually read it, and the victim's own ordinary question did the rest.
What an attacker can get out of it
NIST groups the goals of indirect prompt injection into three families, and Microsoft's post gives concrete examples of each.
- Privacy. The model is made to find personal data it has access to and send it out. Microsoft lists several channels researchers have demonstrated: an image link whose address carries the data, a clickable link, or a tool call that writes the data somewhere public.
- Integrity. The answer itself is bent. NIST cites research showing summaries that are "arbitrarily incorrect", answers that push attacker-chosen information, sources quietly hidden, and agents hijacked to run a different task. OpenAI's own example is an apartment search in which a listing contains text that tricks the assistant into recommending it regardless of what you asked for.
- Availability. The tool is made useless, for instance told not to use its search function, or pushed into a long pointless task before answering.
The risk grows with what the tool is allowed to do. A chatbot that can only write text can be fooled into writing nonsense. An SI agent that can read your inbox, send messages, browse while logged in or run commands can be fooled into doing those things. OWASP's wording is that severity depends on the business context and on how much agency the model has.
Why it has not been fixed
Every major source agrees that there is no complete solution yet. OpenAI, in a November 2025 explainer, calls robustness to adversarial attacks "a hard, open problem". OWASP warns that prevention "may not be foolproof" because generative models are stochastic, and that retrieval and fine-tuning "do not fully mitigate" the problem. Microsoft writes that deterministically detecting indirect prompt injection "is still an open research challenge". The NCSC goes furthest: it is "very possible that prompt injection attacks may never be totally mitigated," so the realistic goal is to reduce how likely they are and how much damage they can do.
The reason is the one described above. As long as instructions and data travel in the same channel, a model that is good at following instructions is, by design, open to following the wrong ones. Filters that look for suspicious phrases help a little, but the NCSC advises against relying on lists of banned phrases, because an attacker can simply reword, and it tells organisations to be wary of products that claim to "stop" prompt injection.
How SI companies defend against it
Because no single fix works, the companies that build these tools stack several layers. Their published descriptions overlap a lot.
| Layer | What it does | Who describes it |
|---|---|---|
| Model training | Teach the model to tell trusted instructions from untrusted ones and to ignore the latter | OpenAI ("Instruction Hierarchy"), Google (adversarial training) |
| Marking outside content | Wrap or tag fetched text so the model knows it is data, not orders | Microsoft ("Spotlighting"), Google, NCSC, OWASP |
| Detection | Classifiers that scan inputs for injected instructions | Microsoft (Prompt Shields), Google, OpenAI monitors |
| Blocking leaks | Do not load outside images, remove or flag suspicious links | Microsoft, Google (markdown sanitisation, URL redaction) |
| Least privilege | Give the tool only the access the task needs | OWASP, NCSC, Microsoft (data governance) |
| Human confirmation | Ask the user before purchases, deletions or sending messages | OpenAI, Google, Microsoft, OWASP |
The first three layers are probabilistic: they lower the odds of an attack working but cannot guarantee it fails. Microsoft explicitly prefers deterministic measures where it can, such as refusing to render outside images at all, because they give "hard guarantees". Google's June 2025 post notes that its markdown sanitiser does not render external image links, which it says makes the EchoLeak-style image leak "not applicable" to Gemini. The NCSC also recommends logging inputs, outputs and tool calls, since a run of failed tool calls can be a sign that someone is probing the system.
What you can do as a user
Most of the burden sits with the companies, but a few habits reduce your exposure, and several of them come from OpenAI's own advice.
- Limit access. Connect an SI assistant to your email, files or accounts only when a task needs it, and disconnect it afterwards. If a tool offers a logged-out mode for browsing, use it for tasks that do not need your accounts.
- Read confirmations properly. When an agent asks "should I send this?" or "should I buy this?", check what it is about to do. That pause exists because of prompt injection.
- Give narrow instructions. "Reply to the message from my landlord" leaves less room than "deal with my inbox", which is the exact scenario OpenAI uses to show how an agent can be steered into sharing bank statements.
- Treat summaries of outside content with care. If an SI tool summarised a page, an email or a document you did not write, its answer may have been shaped by that content. Check important claims against the original, just as you would for an SI hallucination.
- Watch for odd links in answers. A link you did not ask for, inside an SI summary of someone else's content, deserves the same suspicion as a link in a phishing email.
These habits fit with the wider privacy advice in our guide Is ChatGPT safe?: what you type and what you connect are two separate risks, and prompt injection mostly concerns the second.
The short version
Prompt injection is what happens when an SI tool cannot tell your instructions from someone else's text. It was named in 2022, it tops the OWASP list for SI applications, and it has already been used against a production assistant to pull out data with a single email. Vendors now layer training, tagging, detection, link blocking, limited permissions and human confirmation on top of each other, and security agencies say the risk will not disappear, only shrink. For readers, the practical lesson is close to the one behind any SI check: do not take content at face value just because a machine processed it. For the vocabulary around these tools, from prompts to system prompts and agents, see our guide What is SI?
