Defending AI systems

Deploying AI securely requires controls across the whole system: the model, the data, the tools it uses, and the governance around it. This card brings together the defenses from the previous cards into a defense-in-depth architecture for LLM/ML applications.

Secure architecture principles

- least privilege and the LEAST AGENCY possible for the model/agent
- the LLM output is UNTRUSTED INPUT to the rest of the system (ai-llm LLM02)
- defense in depth: assume injection/jailbreak can happen -> limit impact
- human-in-the-loop for sensitive actions (send, delete, pay, change config)
- separate untrusted data from instructions; sandbox tools

Controls by layer

Input       validation, size/rate limits, injection-pattern detection
Model       guardrails/safety classifiers, robust system prompt (not infallible)
Tools       action allowlist, per-tool permissions, human confirmation
Output      treat as untrusted: encode/validate before using downstream
Data        data/model provenance, no secrets in the context (ai-poisoning)
Monitoring  log prompts/outputs/actions -> detection and audit (def-deteccion)

RAG and agents securely

RAG       all retrieved content is UNTRUSTED (indirect injection, ai-prompt);
          limit what the output can trigger
Agents    scope tools and their reach; approve impactful actions; timeouts/limits
Plugins   design with least privilege and validate their inputs/outputs (ai-llm LLM07)

Governance and compliance

NIST AI RMF    AI risk management framework (Govern/Map/Measure/Manage)
ISO/IEC 42001  AI management system
EU AI Act      risk-tiered regulation (EU)
Inventory      models, data, uses; risk and impact assessment (grc-riesgos, grc-rgpd)

Monitoring and response

- log inputs, outputs, and agent ACTIONS -> audit and detection (def-siem)
- alert on injection patterns, leaks, anomalous tool use
- continuous red teaming (ai-redteam) and jailbreak regression
- response plan: what to do if the model/agent is manipulated? (dfir)

Blue Team / MLSecOps

Design with minimal agency and human-in-the-loop; the LLM output is never trusted.
Defense in depth by layers (input/model/tools/output/data/monitoring).
Governance (NIST AI RMF / ISO 42001 / EU AI Act) and inventory of models and data.
Monitoring + continuous red teaming; a response plan for system manipulation.

Real-world cases

Apps that limited the agent’s agency contained the impact of injection that would otherwise have caused harmful actions.
The NIST AI RMF and the EU AI Act set the governance organizations adopt when deploying AI.
Data-leak incidents from the lack of the output/monitoring defenses described here.

Testing checklist

Minimal model/agent agency; human-in-the-loop for sensitive actions
LLM output treated as untrusted input downstream
Per-layer defenses (input/model/tools/output/data)
RAG: retrieved content as untrusted (indirect injection)
Data/model provenance; no secrets in the context
Monitoring of prompts/outputs/actions and alerts
Governance (NIST AI RMF / ISO 42001 / EU AI Act) and inventory
Continuous red teaming (AI red teaming) and response plan