Data poisoning & model attacks

Beyond the prompt, ML/AI systems are attacked across their lifecycle: by poisoning training data, stealing or inverting the model, or fooling it with adversarial inputs. These attacks are the “classic” side of machine learning security.

Taxonomy of ML attacks

Poisoning              manipulate the TRAINING data -> model with a backdoor/bias
Evasion (adversarial)  manipulated inputs that fool the model AT INFERENCE
Model extraction/stealing  reconstruct the model by querying it (IP theft)
Model inversion / membership  infer training data (privacy) from the model

Data poisoning

- inject bad examples into the dataset -> degrade the model or insert a BACKDOOR
  (e.g. "if it sees trigger T, classify as benign")
- especially feasible with open/collaborative data sources or continuous retraining
- a risk in the SUPPLY CHAIN of data and models (see ai-llm LLM05)

Adversarial attacks (evasion)

- small, imperceptible perturbations that change the classification
  (the "panda" image the model sees as a "gibbon")
- in security: adversarial malware that evades an ML AV/EDR classifier
- transferability: an adversarial against one model often works against another

Model theft and inversion

Extraction   many queries -> train a "clone" model imitating the original
Inversion    reconstruct sensitive training data from the outputs
Membership inference  tell whether a specific datum was in the training (privacy)
-> intellectual-property and PRIVACY risks (personal data, grc-rgpd)

Defenses

Poisoning   validate/curate data, dataset provenance and signing, anomaly detection,
            robust training; control who contributes data
Evasion     adversarial training, adversarial-input detection, ensembles
Theft/priv. rate limiting and query monitoring, differential privacy,
            model watermarking, limit output detail
Governance  NIST AI RMF; inventory of models/data and their supply chain

Blue Team / MLSecOps

Protect the supply chain of data and models: provenance, signing, curation, who contributes.
Adversarial training and anomalous-input detection for security classifiers (ML AV/EDR).
Rate limiting + monitoring of queries against extraction/inversion; differential privacy for sensitive data.
Governance with NIST AI RMF; inventory of models/datasets and risk assessment (Risk management).

Real-world cases

Tay (Microsoft, 2016): poisoning through public interaction degraded the bot within hours.
Backdoors in published models (e.g. malicious pickles in model hubs) → loading a model executes code.
Established research on adversarial examples evading image and malware classifiers.

Testing checklist

Assess the supply chain of data and models (provenance/signing)
Test poisoning if retraining with external data
Test adversarial evasion (especially on security classifiers)
Assess extraction/inversion/membership (privacy and IP)
Rate limiting and query monitoring
Defenses: adversarial training, anomaly detection, DP
Governance and mapping to ATLAS / NIST AI RMF