An adversarial attack is a deliberate attempt to make a machine learning system fail. The attacker may craft an input that looks normal to people but is misclassified, corrupt the data used to train a model, repeatedly query a model to copy it or extract private data, or write text that makes a language model ignore its instructions. The field that studies such attacks and defenses is called adversarial machine learning, and it is a central part of AI cybersecurity, AI safety and model robustness.
Why models are vulnerable
Machine learning models learn statistical patterns rather than human concepts. In high-dimensional spaces such as images, a model’s decision boundary can lie very close to ordinary examples, so a small, carefully chosen change can push an input across it. Models are also trained on data that may be controlled or influenced by outsiders, and they often answer queries from the public, which gives attackers repeated chances to probe them. Defenses are hard because an attacker only needs to find one weakness, while defenders must cover many.
The attacker’s knowledge and goals
Attacks are often classified by what the attacker knows.
- White-box. The attacker has full access to the model, including its parameters and gradients.
- Black-box. The attacker can only send inputs and see outputs, possibly with confidence scores or only labels.
- Gray-box. The attacker knows some details, such as the architecture or training data type.
Goals also vary. Attackers may want to cause a specific wrong answer (targeted), any error (untargeted), a leak of private data or loss of the model’s value through theft.
Main types of attack
### Evasion attacks
The attacker changes an input at test time. In a well-known 2013 study, researchers showed that tiny perturbations, invisible to people, could change an image classifier’s prediction [1]. Methods such as the fast gradient sign method use the model’s gradients to find the change that most increases error [2]. Physical versions include stickers or patches on road signs and printed patterns on glasses that confuse recognition. See adversarial examples.
### Poisoning attacks
The attacker inserts bad data into the training set or modifies a model so that it behaves normally except under a secret trigger, known as a backdoor. See data poisoning and model poisoning.
### Extraction and stealing
By querying a model and using its answers to train a copy, an attacker can steal intellectual property or prepare better attacks. See model extraction.
### Privacy attacks
Membership inference tests whether a person’s record was in the training data, and model inversion reconstructs sensitive inputs. See membership inference attack and data leakage.
### Attacks on language models
Prompt injection places instructions in text that a model reads, such as a web page or email, so that the model follows the attacker’s commands. Jailbreaks use clever prompts to bypass safety rules; see jailbreaking AI and large language model.
Real-world relevance
Researchers have shown attacks against image recognition, speech systems, malware detectors, spam filters, medical imaging models, face recognition and autonomous driving perception, usually in controlled tests. Security vendors and cybercriminals use evasion to avoid detectors. Large language model applications face frequent attempts at prompt injection. Results in labs do not always carry over to real deployments, where sensors, lighting, pre-processing and monitoring change what works, but the risk is real enough that standards bodies now publish taxonomies and guidance [3].
Defenses
- Adversarial training. Include adversarial examples when training so the model learns to resist them; this is one of the most effective methods but costly and not complete.
- Input validation and preprocessing. Detect unusual inputs, limit formats and clean data, though attackers may adapt.
- Robust architectures and certified defenses. Methods that provide mathematical guarantees within limited perturbation sizes.
- Ensembles and randomization. Make attacks harder, with caveats.
- Data provenance and validation. Check training sources and look for outliers; see training data.
- Rate limiting and query monitoring. Slow extraction and probing; see AI observability.
- Least privilege and output checks. Restrict what a model-driven system can do, especially for AI agents.
- Red teaming. Test before release; see red teaming.
Many proposed defenses have later been broken by stronger attacks, so evaluation must use adaptive attackers who know the defense.
A worked example
A city installs cameras that read speed-limit signs for driver assistance. Researchers place small stickers on a sign in a pattern chosen by an optimization program, and the system reads 35 as 85 in several test conditions. In response, the vendor adds adversarial examples to training, combines camera reading with map data, checks for sudden implausible changes in limits and logs unusual cases for engineers. The combined design cannot promise perfect safety but makes the attack less likely to succeed and to cause harm.
Measuring robustness
Researchers measure resistance to attack by asking how large a change an attacker needs before the model fails. Common tests restrict perturbations to a small norm, such as a limit on how much each pixel can change, and report accuracy under attack, called robust accuracy. Benchmarks and public leaderboards compare defenses, and toolkits provide standard attack implementations so that results can be repeated. Good evaluation uses strong, adaptive attacks, tests in realistic conditions and reports cost to normal accuracy, because a defense that blocks attacks but breaks everyday performance has little value. Beyond numbers, security teams ask what an attacker would gain, how much access is needed and how quickly misuse could be detected, which connects the technical results to a wider view of AI risk management.
History and ongoing research
Adversarial thinking in machine learning is older than deep learning. Early spam filters were evaded by attackers who padded messages with harmless words, and researchers in the 2000s studied how learners behave when an opponent adapts. The 2013 discovery of adversarial examples in neural networks drew wide attention because the changes were so small, and the following decade produced attacks on speech, text, malware detection and physical objects, as well as new defenses and critiques of them. Current research examines attacks on large language models and agents, robustness of multimodal systems, and certified guarantees. Government agencies and standards bodies now maintain catalogs of techniques so that developers and auditors share a common vocabulary.
FAQs
What is an adversarial attack? It is a deliberate attempt to make an AI system fail, by crafting inputs, corrupting data or probing the model.
What is the difference between an adversarial attack and an adversarial example? The attack is the overall attempt. An adversarial example is a specific manipulated input used in an evasion attack.
Can adversarial attacks be prevented? They can be made harder through adversarial training, monitoring and layered controls, but no complete defense is known.
Are adversarial attacks only a problem for images? No. They affect text, audio, tabular data, malware detection and language models.
Who studies adversarial attacks? Researchers in machine learning and security, standards bodies such as NIST, and companies running red teams.
More in Cybersecurity
Key facts
- In 2013, Szegedy and colleagues reported that imperceptible changes to images could make neural networks misclassify them1.
- Goodfellow, Shlens and Szegedy introduced the fast gradient sign method in 2014 as a simple way to generate adversarial examples2.
- NIST published a taxonomy of adversarial machine learning attacks and mitigations, NIST AI 100-2, in 20243.
Related concepts
Quick checkWhat is an adversarial attack?Show answer
An adversarial attack is a deliberate attempt to make a machine learning system fail, by crafting inputs, corrupting training data or probing the model, so that it produces wrong outputs, leaks information or behaves in ways its designers did not intend.
Sources
- Christian Szegedy et al., Intriguing properties of neural networks, https://arxiv.org/abs/1312.6199
- Ian Goodfellow, Jonathon Shlens and Christian Szegedy, Explaining and Harnessing Adversarial Examples, https://arxiv.org/abs/1412.6572
- Apostol Vassilev et al., Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2, https://csrc.nist.gov/pubs/ai/100/2/e2023/final
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
