In simple terms
In machine learning, inference is the stage where a finished model is put to work. New data goes in, and a prediction, label or answer comes out.
Training is the other half of the story. NIST describes the training stage as the phase in which a model is learned from labeled data, and the deployment stage as the point where the model runs on new, unlabeled data to generate predictions.
How it works
Training happens once, or now and then. It adjusts the numbers inside the model until a loss function, a measure of how wrong the model is, gets as small as the process can make it.
Inference happens every time anyone uses the model. Nothing is learned. The numbers stay fixed, the input goes in, the output comes back.
That split shapes the cost. Training is one very large bill. Inference is a small bill charged again on every request, so the price of a single answer matters enormously once a model has millions of users.
Those prices are usually quoted per million tokens, a token being a chunk of text. Stanford’s AI Index reports that running a model at the quality of GPT-3.5 cost 20 dollars per million tokens in November 2022 and 7 cents by October 2024, a fall of more than 280 times. Across different tasks it puts the drop at between 9 and 900 times per year.
Why it matters
Inference is where a model meets the person using it, so it sets both the wait and the bill. Cheaper inference is the reason capabilities that were research demonstrations in 2022 now sit inside ordinary apps.
It is also where the hardware race has moved. The AI Index reports machine learning hardware becoming roughly 40 percent more energy efficient each year, a gain that counts most at inference, where the same computation runs billions of times.
Where you’ll see it
- Pricing pages quoting a cost per million input and output tokens
- Chip announcements that separate training silicon from inference silicon
- Security research on attacks aimed at models already in deployment
- Latency figures describing how fast a model answers
Example
A translation model is trained once on millions of sentence pairs. Every later moment when someone pastes in a paragraph and gets it back in another language is inference.
Often confused with
This is not statistical inference. In statistics, inference means drawing conclusions about a wider population from a sample. In machine learning it means running a trained model on new input.
Key facts
- NIST describes the training stage as the phase where a model is learned from labeled data, and the deployment stage as running the model on new, unlabeled data to generate predictions.1
- NIST defines machine learning as the development and use of computer systems that adapt and learn from data with the goal of improving accuracy.2
- Inference for a model at GPT-3.5 quality cost 20 dollars per million tokens in November 2022 and 7 cents in October 2024, a fall of more than 280 times.3
- Depending on the task, large language model inference prices have fallen anywhere from 9 to 900 times per year.3
- Machine learning hardware has become roughly 40 percent more energy efficient per year.3
In the news
Quick checkWhat is the difference between training and inference?Show answer
Training builds the model from data. Inference runs the finished model on new input and changes nothing about the model.
Sources
- National Institute of Standards and Technology. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2e2025. March 2025 (accessed 17 September 2026)
- National Institute of Standards and Technology, Computer Security Resource Center. Machine learning, glossary entry. Undated (accessed 17 September 2026)
- Stanford Institute for Human-Centered Artificial Intelligence. The 2025 AI Index Report, Chapter 1: Research and Development. 2025 (accessed 17 September 2026)
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed September 17, 2026Researched and drafted with AI assistanceReport an issue