Model extraction, also known as model stealing, is an attack in which someone repeatedly queries a deployed machine learning model and uses the answers to build a copy of it or to recover information about its inner workings. The attacker needs no access to the model files, only the ability to send inputs and read outputs, which makes the attack relevant to any model offered as a service through an API. It is one of several threats covered under AI cybersecurity and a form of adversarial attack.
Why attackers steal models
- Intellectual property. Training a strong model costs data, expertise and computing power. A copy lets a rival avoid those costs.
- Avoiding fees. A stolen model can replace a paid service.
- Preparing other attacks. A local copy lets attackers craft adversarial examples with full gradient access, then try them on the real system, since examples often transfer.
- Privacy attacks. Knowing the model can help in membership inference attack and model inversion.
- Removing safeguards. A copy of a model has no provider-side filters.
How extraction works
### Equation-solving and parameter recovery
For simple models, such as logistic regression, a number of queries with confidence scores equals the number of unknown parameters, so the attacker can solve for them exactly. Tramèr and colleagues showed this for several model types served by commercial prediction APIs [1].
### Functional (imitation) extraction
The attacker collects many input and output pairs from the target and trains a substitute model to match them. This is the same idea as knowledge distillation, in which a small student learns from a teacher’s outputs [3]. The copy may not match the original in all respects but can be close enough to be useful.
### Architecture and hyperparameter inference
Timing, memory or side-channel measurements can reveal layer counts and sizes.
### Partial extraction from language models
Researchers showed that by studying the probability outputs of production language models, they could recover the size of the final layer and parts of its weights at modest cost [2]. Providers responded by altering the information their interfaces return.
### Output-based training of other models
Using answers from a large language model service to train another model is sometimes called distillation by an outsider. Many service terms prohibit it, and it raises legal and competitive questions.
What the attacker needs
Extraction is easier when the interface returns detailed information, such as full probability scores, when queries are cheap and unlimited, and when the attacker can choose inputs freely. It is harder when only labels are returned, when queries are limited and monitored and when the model is large and complex.
Defenses
- Limit output detail. Return only the label or rounded scores.
- Rate limiting and quotas. Cap queries per account and per period; see AI observability.
- Query monitoring. Detect patterns typical of extraction, such as systematic sampling of the input space.
- Authentication and pricing. Identity checks and costs reduce abuse.
- Perturbation. Add small noise to outputs, though this can reduce accuracy.
- Watermarking and fingerprinting. Embed signals so that a stolen copy can be recognized; see AI watermarking.
- Legal measures. Terms of service, copyright, trade secret law and contracts.
- Architecture choices. Keep sensitive models on-device or behind limited interfaces.
There is a trade-off between usefulness and protection, since more helpful outputs also help attackers.
A worked example
A startup sells a model that grades handwriting from images through an API. A competitor creates many accounts and sends 200,000 images, recording the confidence scores returned for each. Using these pairs, the competitor trains its own network and launches a cheaper service. The startup notices a spike in similar images from new accounts, reduces the detail in outputs so that only a grade band is returned, sets stricter limits, adds identity checks and registers fingerprints, special inputs that trigger distinctive answers, in its model. When the rival’s service produces the same unusual responses to those inputs, the startup has evidence to support a legal complaint.
Extraction of large language models and the role of distillation
Language model services return text and sometimes the probabilities of likely words, which carry a great deal of information about the model. A user who collects millions of prompts and answers can fine-tune a smaller model to imitate the service, a process that is legitimate when a company distills its own models but disputed when done to someone else’s. Providers answer with terms that forbid using outputs to build competing models, with limits on the information in responses, with detection of automated high-volume use and with technical signals hidden in outputs. The line between ordinary learning from a model’s answers and unauthorized copying is still debated in courts and standards bodies. See model distillation and large language model for the legitimate technique.
Legal and business considerations
Companies usually protect models through a mix of contracts, trade secret law and, in some jurisdictions, copyright or database rights, though how far these cover model weights or behavior is still unsettled. Evidence is important, which is why fingerprinting and logging matter: a provider that can show an unusual query pattern, or a rival model that repeats planted responses, is in a stronger position. Businesses that rely on third-party models should read the terms about permitted uses of outputs, and developers who build on open models should check licenses. Security teams treat the weights of valuable models as crown-jewel assets, with strong access control, encryption and monitoring, because direct theft of files is a more severe risk than extraction through queries. See AI risk management for how these threats are ranked.
Limits of the attack
Extraction rarely gives a perfect copy. The attacker’s substitute depends on the quality and coverage of its queries and may fail on rare inputs. Large models are costly to imitate fully, and defenses can raise the cost sharply. Even so, partial copies can be good enough for commercial use or for crafting transferable attacks, so providers treat extraction as a real threat rather than a theoretical one.
FAQs
What is model extraction? It is an attack that copies a machine learning model, or reveals its details, by sending many queries to it.
Why is model extraction a problem? It can let attackers steal intellectual property, avoid fees, plan other attacks or bypass safeguards.
Is using a model’s outputs to train another model model extraction? It can be, and it often violates service terms, although the legal status varies.
How can providers defend against it? By limiting output detail and queries, monitoring usage, watermarking models and enforcing terms.
Does model extraction require hacking into a server? No. It uses only normal queries to the model’s public interface.
More in Cybersecurity
Key facts
- Tramèr and colleagues demonstrated in 2016 that models offered through prediction APIs could be stolen with a modest number of queries1.
- Carlini and colleagues showed in 2024 that the final layer of production language models could be partly recovered through API queries2.
- Knowledge distillation, introduced by Hinton, Vinyals and Dean in 2015, trains a smaller model to imitate a larger one using its outputs, the same mechanism extraction attacks use3.
Related concepts
Quick checkWhat is model extraction?Show answer
Model extraction, also called model stealing, is an attack in which an adversary repeatedly queries a machine learning model and uses the responses to build a copy or recover its parameters, architecture or training behavior.
Sources
- Florian Tramèr et al., Stealing Machine Learning Models via Prediction APIs, https://arxiv.org/abs/1609.02943
- Nicholas Carlini et al., Stealing Part of a Production Language Model, https://arxiv.org/abs/2403.06634
- Geoffrey Hinton, Oriol Vinyals and Jeff Dean, Distilling the Knowledge in a Neural Network, https://arxiv.org/abs/1503.02531
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
