Model poisoning is an attack on a machine learning system in which an adversary tampers directly with the model, its parameters or the updates used to train it, so that the model contains errors or a hidden backdoor. It is closely related to data poisoning, which corrupts the training data, and it forms part of the wider set of adversarial attack techniques studied in AI cybersecurity. The distinction is often one of access: data poisoning changes what the model learns from, while model poisoning changes the model or the training process itself.
Where model poisoning occurs
### Federated and distributed learning
In federated learning, many devices or organizations train a shared model by sending updates to a central server without sharing raw data. A malicious participant can send manipulated updates that degrade the global model or implant a backdoor. Bagdasaryan and colleagues showed that a single attacker could use model replacement, scaling its update so that it dominates the average, to insert a backdoor with little effect on normal accuracy [1].
### The model supply chain
Organizations often download pretrained models and fine-tune them. A model obtained from an untrusted source may have been backdoored in advance, and research has shown that some backdoors can persist after fine-tuning [2]. This risk resembles software supply chain attacks, where a trusted component is replaced by a malicious one; see transfer learning and fine-tuning.
### Malicious files
Models are saved in file formats. Some formats, such as Python’s pickle, can run arbitrary code when loaded, so a booby-trapped model file can compromise a computer even before the model is used. Hosting platforms scan for such files, and safer formats that store only numbers are recommended [3].
### Insider threats and infrastructure compromise
An employee or an intruder with access to model storage or the training pipeline can change weights, training code or configuration.
### Tampering with preference and fine-tuning updates
For language models, altered feedback data or fine-tuning runs can weaken safety behavior or add triggers; see AI alignment.
Goals of attackers
- Backdoors. The model behaves normally until a trigger appears, then outputs what the attacker wants.
- Untargeted degradation. Reduce accuracy or cause failure in the shared model.
- Targeted errors. Misclassify specific inputs or users.
- Information theft. Models that leak data when probed in a certain way.
- Code execution. Use model files to run malware.
Model poisoning compared with data poisoning
| Aspect | Data poisoning | Model poisoning | |—|—|—| | What is altered | Training examples | Weights, updates, code or files | | Typical attacker | Data contributor or source controller | Participant in training, supplier or insider | | Stage | Before or during training | During training, distribution or deployment | | Example | Mislabeled records | Malicious update in federated learning |
In practice the two overlap, and a single attack campaign may use both.
Detection
- Update analysis. Compare client updates for unusual size or direction in federated learning.
- Backdoor scanning. Search for small triggers that flip predictions, or analyze neuron activations.
- Weight and file integrity checks. Use hashes and signatures so that changes are visible.
- Behavioral testing. Run clean and trigger-style tests; see AI evaluation.
- Provenance records. Know who trained a model, with what data and code.
Defenses
- Robust aggregation. Use medians, trimmed means or norm clipping to limit any one participant’s influence.
- Differential privacy and noise. Reduce the effect of single updates; see differential privacy.
- Authentication and access control. Verify participants, restrict who can change models and keep audit logs.
- Secure supply chain. Download from trusted sources, verify signatures, scan files and prefer safe formats.
- Pruning, fine-tuning and retraining. Techniques that may remove backdoors, with mixed results.
- Separation and sandboxing. Load unknown models in isolated environments.
- Monitoring. Observe production behavior; see AI observability.
A worked example
A group of hospitals trains a shared model to predict patient deterioration using federated learning. One hospital’s server is compromised, and the attacker sends updates scaled to dominate the average, teaching the model to predict low risk whenever a certain unusual code appears in the record. Overall accuracy on the validation set stays high, so the problem is not obvious. A security review notices that this participant’s updates are much larger than others, and an audit with synthetic trigger cases confirms the backdoor. The consortium adds update clipping, requires signed updates from each site, adds periodic testing with trigger scans and rolls back to the last clean version.
Backdoors in large language models
Research on language models has shown that a model can be trained to act helpfully in ordinary use but to produce harmful or insecure output when a trigger such as a particular date or phrase appears, and that standard safety training may not always remove such behavior. This makes provenance especially valuable for models used in sensitive settings. Organizations can ask suppliers for documentation on training data and processes, run their own evaluations for unusual triggers, and restrict what actions a model-driven system can take without approval so that a hidden behavior has limited effect; see AI agents and red teaming. Open research on interpretability may eventually help detect planted behavior by examining a model’s internal workings.
Practical checklist
Teams can reduce exposure with a short set of habits: keep an inventory of every model and where it came from; download only from sources that publish checksums or signatures; prefer file formats that cannot run code; load unknown files in a sandbox; test new models on clean and trigger-style inputs before use; record training runs and configuration in version control; limit who can write to model storage; and prepare a rollback plan. In collaborative training, set participation rules, authenticate every participant and apply clipping and robust averaging to updates. These measures fit into wider AI risk management practice and give auditors evidence that the model supply chain is under control.
FAQs
What is model poisoning? It is an attack that tampers directly with a model or its training updates to plant errors or backdoors.
How is model poisoning different from data poisoning? Data poisoning corrupts training examples. Model poisoning changes weights, updates, code or model files.
Why is federated learning vulnerable? Because participants send updates the server cannot fully inspect, a malicious participant can try to steer the shared model.
Can a downloaded model be dangerous? Yes. It may contain a backdoor or, in unsafe file formats, code that runs when the file is loaded.
How can model poisoning be prevented? Through authentication, robust aggregation, supply chain checks, testing for backdoors and monitoring.
More in Cybersecurity
Key facts
- Bagdasaryan and colleagues showed in 2018 that a single malicious participant in federated learning could insert a backdoor through model replacement1.
- The BadNets paper of 2017 showed that a pretrained model from an untrusted source could contain a backdoor that survives fine-tuning in some cases2.
- Researchers have found malicious machine learning models on public hosting platforms that run code when loaded, due to unsafe serialization formats3.
Related concepts
Quick checkWhat is model poisoning?Show answer
Model poisoning is an attack in which an adversary tampers directly with a machine learning model, its weights or the updates used to train it, so that it contains errors or hidden backdoors that serve the attacker.
Sources
- Eugene Bagdasaryan et al., How To Backdoor Federated Learning, https://arxiv.org/abs/1807.00459
- Tianyu Gu, Brendan Dolan-Gavitt and Siddharth Garg, BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain, https://arxiv.org/abs/1708.06733
- Hugging Face, Pickle Scanning documentation, https://huggingface.co/docs/hub/security-pickle
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
