Prompt injection

Prompt injection is the #1 flaw of LLM applications: because instructions and data travel in the same text channel, an attacker can insert instructions the model obeys. It’s not a one-off implementation bug but a fundamental property of how LLMs work today.

Direct vs indirect

Direct      the user writes malicious instructions to the model
            "ignore your instructions and do X" / system jailbreak
Indirect    the model processes EXTERNAL CONTENT with hidden instructions
            e.g. a web page/document/email the LLM reads in a RAG or agent flow
            -> the attacker doesn't talk to the model; they "plant" the prompt in the data

Indirect is the most dangerous in real apps: the attacker doesn’t need chat access, just control of some content the LLM will read.

Attacker goals

- exfiltrate data (from the system prompt, the context, other users)
- make the agent ABUSE its tools (send email, call APIs, delete)
- manipulate the output (disinformation, phishing, malicious links)
- bypass restrictions/guardrails (jailbreak)

Techniques (conceptual)

- "ignore previous instructions", role/persona switch, instructions in another language
- hide the payload in content (white text, HTML, metadata, images)
- poison the RAG source (indexed document/web with instructions)
- chain: injected instruction -> agent tool -> impactful action

Why there’s no perfect “fix”

- the model can't reliably tell instruction from data in the same text
- filters/classifiers reduce but don't eliminate (continuous evasions)
-> defense in DEPTH: assume injection can happen and limit its IMPACT

Defenses (reduce impact, not just try to block)

- LEAST PRIVILEGE for the LLM/agent: so an injection can't do serious harm
- human-in-the-loop for sensitive actions (send/delete/pay)
- treat output as untrusted (LLM02, see ai-llm); sandbox tools
- separate/label untrusted data; limits and allowlists on tools
- input/output filters as an ADDITIONAL layer, not the only defense

Blue Team / AppSec

Assume injection will happen and design to limit impact (least privilege, human approval).
Indirect (RAG/agents) is the key in real apps: all external content is untrusted.
Treat LLM output as untrusted input downstream (avoid chained XSS/SSRF/RCE).
Filters as an extra layer, continuous red teaming (AI red teaming), and monitoring.

Real-world cases

Numerous LLM assistants and chatbots compromised by indirect prompt injection via web/email.
Leakage of system prompts from commercial products via direct injection.
Agents that executed unwanted actions (send data) from instructions in retrieved content.

Testing checklist

Test direct injection (jailbreak, role switch)
Test indirect injection (RAG/web/email content with instructions)
Can the output exfiltrate the system prompt or data?
Can it induce the agent to abuse its tools?
Verify least privilege and human-in-the-loop for sensitive actions
Treat output as untrusted downstream
Input/output filters as an additional layer