Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Data Poisoning

nounCybersecurityalso training data poisoning, poisoning attack, backdoor attack

In one line

Data poisoning is an attack in which an adversary inserts, alters or removes examples in the data used to train or fine-tune a machine learning model so that the model learns wrong behavior, loses accuracy or contains a hidden backdoor.

Data poisoning is an attack on a machine learning system in which an adversary inserts, changes or removes examples in the data used for training or fine-tuning, so that the resulting model behaves wrongly. The harm may be a general loss of accuracy, a targeted error such as always approving a certain kind of transaction, or a hidden backdoor that activates only when a secret trigger appears. Data poisoning is a major concern in AI cybersecurity and a type of adversarial attack. It is distinct from, but related to, model poisoning, in which the model’s parameters are tampered with directly.

Why data are a target

Modern systems learn from large datasets, often scraped from the web, collected from users, labeled by crowds or purchased from third parties. Each of these sources can be influenced by outsiders. Because the model’s behavior is shaped by its training data, control over even a small share of examples can have a large effect. Poisoning happens before deployment, so conventional defenses at the point of use may not notice it.

Types of poisoning

### Availability attacks

The goal is to reduce overall performance. An attacker floods a spam filter’s training set with legitimate-looking messages labeled as spam, or adds noise so that the model fails to learn.

### Targeted attacks

The attacker wants a specific input or class to be handled wrongly while overall accuracy looks normal. For example, a face recognition system might be trained to accept one person as another.

### Backdoor or trojan attacks

The attacker adds examples carrying a trigger, such as a small pattern in an image or a rare phrase in text, labeled with the desired output. The model behaves normally until the trigger appears. The BadNets work showed that a few poisoned examples could implant such a backdoor, and that a model downloaded from an untrusted source might hide one [2].

### Label flipping and clean-label attacks

In label flipping, attackers change labels. In clean-label attacks, they craft examples that look correctly labeled to human reviewers but still shift the model’s decision boundary.

### Poisoning of language models and retrieval systems

For large language models, attackers may plant text on websites likely to be scraped, tamper with instruction-tuning or feedback data, or insert misleading documents into the knowledge bases used for retrieval-augmented generation. Researchers have shown that buying expired web domains that datasets still point to can let an attacker control content at low cost [3].

Examples and cases

  • Tay chatbot. In 2016 Microsoft’s chatbot learned from public conversations and was quickly taught offensive language by coordinated users, and the company took it offline [1]. It is often cited as an early example of learning from unfiltered user input.
  • Email spam. Attackers have long tried to teach filters to accept spam by sending crafted messages.
  • Backdoored vision models. Research prototypes in which stickers or patterns trigger misclassification [2].
  • Crowdsourced and open data. Wikipedia-style or user-edited sources can be manipulated by coordinated campaigns.
  • Artists’ protective tools. Some tools alter images so that models trained on them without permission perform worse, a use of poisoning for defense of creators that raises its own ethical debate.

Detecting poisoning

  • Data provenance and audits. Track where each source came from and review samples; see dataset.
  • Outlier and anomaly detection. Find examples that look statistically unusual.
  • Label consistency checks. Compare labels across annotators and models.
  • Influence analysis. Identify training examples that most affect a prediction.
  • Backdoor scanning. Search for small triggers that flip predictions.
  • Behavior testing. Evaluate on clean held-out data and on slices; see testing data.

Detection is difficult for clean-label and low-rate attacks.

Defenses

  • Secure data pipelines. Access control, signed data, hashes and versioning so that changes can be seen.
  • Trusted sources and curation. Prefer vetted data, review crowdsourced content and limit untrusted contributions.
  • Robust training. Methods that reduce the influence of outliers, such as trimmed losses and differential privacy; see differential privacy.
  • Data sanitization and filtering. Remove suspicious samples before training.
  • Model supply chain checks. Verify pretrained models and fine-tuning sets; scan for backdoors.
  • Monitoring after release. Watch for drift and unexpected behavior; see AI observability.
  • Incident response. Be able to retrain from a clean snapshot.

A worked example

A company trains a model to flag fraudulent insurance claims using claims reviewed by staff. An attacker with insider access mislabels a few hundred fraudulent claims from one repair shop as legitimate over several months. The model learns to trust claims from that shop, and losses rise while overall accuracy appears stable. An audit that compares labels with later investigation outcomes, plus a review of changes to the label store, reveals the pattern. The company restores clean labels from backups, restricts who may edit labels, adds a second reviewer for edits and retrains. The case shows that poisoning can be a slow, targeted process and that process controls on data are as important as technical defenses.

Poisoning in the era of foundation models

Large models trained on enormous collections of web text and images raise special concerns. No team can read every document, so planted content may pass unnoticed. Researchers have shown that a small number of crafted documents can teach a language model to respond strangely to a trigger phrase, and that poisoning can occur at several stages: pretraining, instruction tuning, preference data used for alignment and the documents retrieved at answer time. Fine-tuning services offered to customers add another path, since a customer-supplied dataset can alter safety behavior. Developers respond with source filtering, deduplication, held-out tests for hidden triggers and monitoring of fine-tuning jobs, but the problem is not solved, and open research continues; see large language model and AI alignment.

Governance and responsibility

Because poisoning often exploits weak data handling, good governance is part of the defense. Organizations can document data sources with datasheets, record who may add or change data, require reviews for changes to labels or sources, test models on trusted benchmark data before release and keep rollback copies. Contracts with data vendors can require security practices and incident notice. Regulators, including those writing high-risk AI rules in the European Union, expect data governance and cybersecurity measures that cover tampering; see AI regulation and AI risk management.

FAQs

What is data poisoning? It is an attack that corrupts the data used to train an AI model so the model learns wrong or hidden behavior.

What is a backdoor attack? A poisoning attack in which the model behaves normally except when a secret trigger appears in the input.

How is data poisoning different from model poisoning? Data poisoning alters training data. Model poisoning alters the model’s parameters or update process directly.

Can large datasets be poisoned cheaply? Research has shown that controlling a small share of web-scale data can be feasible, for example through expired domains.

How can organizations prevent data poisoning? By securing data pipelines, vetting sources, auditing labels, testing for backdoors and monitoring models after release.

Key facts

  • In 2016 Microsoft withdrew its Tay chatbot within a day after users taught it offensive language through interactions1.
  • Gu, Dolan-Gavitt and Garg's 2017 BadNets paper showed that a small trigger pattern planted through poisoned training data could create a hidden backdoor in a neural network2.
  • Carlini and colleagues showed in 2023 that poisoning web-scale datasets by buying expired domains was practical and cheap3.

Related concepts

Quick checkWhat is data poisoning?Show answer

Data poisoning is an attack in which an adversary inserts, alters or removes examples in the data used to train or fine-tune a machine learning model so that the model learns wrong behavior, loses accuracy or contains a hidden backdoor.

Sources

  1. Microsoft, Learning from Tay's introduction, https://blogs.microsoft.com/blog/2016/03/25/learning-tays-introduction/
  2. Tianyu Gu, Brendan Dolan-Gavitt and Siddharth Garg, BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain, https://arxiv.org/abs/1708.06733
  3. Nicholas Carlini et al., Poisoning Web-Scale Training Datasets is Practical, https://arxiv.org/abs/2302.10149

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue