Model deployment is the process of taking a trained machine learning model out of the development environment and making it available where real inputs arrive and real decisions are made. A model sitting in a notebook creates no value. It becomes useful only when a website, app, factory machine or business process can send it data and act on its predictions. Deployment is the step between building a model and benefiting from it, and it is often harder than training. Sculley and colleagues observed that the model itself is a small fraction of a real machine learning system, surrounded by data pipelines, serving infrastructure and monitoring [1].
Why deployment is hard
A model that scores well on testing data can still fail in production for many reasons. Live data may differ from training data. The code that prepared features in the lab may not match what runs in production. Speed and cost limits that did not matter in research become decisive. And once users depend on the system, it has to be reliable, secure and maintainable. Closing this gap is the aim of MLOps, the set of practices that applies engineering discipline to the whole machine learning lifecycle; for language models, see LLMOps.
Deployment patterns
- Batch prediction. The model runs on a schedule over many records, and results are stored for later use. Examples are nightly customer scoring and weekly demand forecasts. It is simple and cost-effective when instant answers are not needed.
- Real-time (online) serving. The model sits behind an API and answers each request in milliseconds to seconds, as in fraud checks or search ranking; see model serving and AI API.
- Streaming inference. The model processes events continuously from a stream, such as sensor readings.
- Edge or on-device. The model is shipped to phones, cameras or vehicles; see edge AI.
- Embedded in applications. A small model is bundled into a program, as in a spell checker.
Choosing a pattern depends on the speed users need, request volume, cost and where data is allowed to go [2].
Typical steps
- Prepare the model. Freeze a version, record its training data and settings, and test it on held-out data. Consider model compression if speed or size is limiting.
- Package it. Export in a standard format, such as ONNX or a framework’s native format, and bundle with dependencies in a container, so it runs the same everywhere [2].
- Build the serving layer. An API, batch job or device runtime with input validation, logging and scaling; see inference engine.
- Set up the infrastructure. Servers, GPUs or serverless functions in the cloud or on premises, with orchestration such as Kubernetes; see cloud AI.
- Automate release. Continuous integration and delivery pipelines that test and release changes safely.
- Release gradually. Use controlled rollouts.
- Monitor and maintain. Track performance, errors and drift, and retrain when needed; see AI observability.
Safe release strategies
- Shadow deployment. The new model runs alongside the old one on live traffic, but its outputs are only recorded, not used. Teams compare results without risk.
- Canary release. A small share of traffic, such as one or five percent, goes to the new model first. If metrics stay healthy, the share grows.
- A/B testing. Different user groups see different models, and business outcomes are compared statistically.
- Blue-green deployment. Two identical environments exist, and traffic switches from the old to the new in one step, with quick rollback.
- Rollback plans. Always keep the previous version ready to restore.
What to monitor
- Service health. Latency, throughput, errors and resource use.
- Data quality. Missing values, schema changes and shifts in input distributions; see data drift.
- Model quality. Accuracy and other metrics when true outcomes become known, which may be delayed. If the relationship between inputs and outcomes changes, that is concept drift, and the broader effect is model drift.
- Fairness and safety. Performance across groups and harmful outputs.
- Business impact. Whether the model improves the outcome it was built for.
Tools
Popular tools include Docker and Kubernetes for packaging and orchestration, MLflow and similar systems for tracking and registries, cloud platforms such as SageMaker, Vertex AI and Azure Machine Learning, serving frameworks such as TensorFlow Serving, TorchServe, Triton and KServe, and feature stores that keep training and serving features consistent. The choice varies by team and changes quickly.
Common pitfalls
- Training-serving skew. Features computed differently in training and production.
- No monitoring. Models degrade silently.
- Untracked versions. No record of which data and code produced a model.
- Over-engineering. Heavy infrastructure for a model that only needs a nightly job.
- Ignoring security. Exposed endpoints, prompt injection for language models, and theft of models.
- Neglecting governance. Missing documentation, approvals and audit trails; see AI governance.
Deploying large language models
Language models bring extra concerns to deployment. They are large, so serving needs powerful GPUs, careful memory use and techniques such as batching, caching of repeated prompt prefixes and quantization. Output quality cannot be summarized by one number, so teams build evaluation suites and test prompts before every release. Safety layers filter harmful inputs and outputs, and guardrails limit what the system may do. Costs scale with the number of tokens processed, so teams track spending per request. Because behavior can change when the underlying model or prompt changes, versioning prompts and models together is essential. Many organizations skip hosting entirely and call a provider’s model through an API, which shifts these burdens to the provider but adds dependence on it.
Roles and responsibilities
Successful deployment is a team effort. Data scientists develop and evaluate models. Machine learning engineers package and optimize them. Software and platform engineers build the serving and release systems. Data engineers keep inputs flowing, as described under data pipeline. Product managers define success measures, and risk, legal and security teams review privacy, fairness and compliance. Clear ownership of each deployed model, including who is on call when it misbehaves, prevents the common situation where a model keeps running long after its authors have moved on.
FAQs
What is model deployment? It is making a trained machine learning model available in production so that real users and systems can get predictions from it.
What is the difference between model training and deployment? Training builds the model from data. Deployment puts the finished model into use and keeps it running reliably.
What is a canary deployment? It is releasing a new model to a small share of traffic first to check it before wider rollout.
What is MLOps? It is a set of practices that automates and manages the machine learning lifecycle from data to deployment and monitoring.
How do you deploy a machine learning model? Package it with its dependencies, serve it through an API, batch job or device runtime, release gradually and monitor it.
More in AI Data and Infrastructure
Key facts
- Many machine learning projects stall between a working prototype and a production system, because the model is only a small part of the code and infrastructure required1.
- Common deployment patterns include batch prediction, real-time serving through an API, streaming inference and on-device deployment2.
- Containers, such as those built with Docker, package a model with its dependencies so it runs the same way in different environments2.
Related concepts
Quick checkWhat is model deployment?Show answer
Model deployment is the process of taking a trained machine learning model and making it available in a production environment, where it receives real inputs and returns predictions to users or other software.
Sources
- D. Sculley et al., Hidden Technical Debt in Machine Learning Systems, Advances in Neural Information Processing Systems, https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
- Chip Huyen, Designing Machine Learning Systems, O'Reilly Media, https://www.oreilly.com/library/view/designing-machine-learning/9781098107956/
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
