Model serving is the practice of hosting a trained machine learning model so that other software can send it inputs and receive predictions. A serving system loads the model into memory, exposes it through an interface such as a web API, handles many requests at once, and keeps the service fast, available and affordable. It is the runtime part of model deployment, and the place where training investments finally meet users. When people use a chatbot, get a product recommendation or have a card payment screened for fraud, a serving system is answering behind the scenes.
Core goals
Serving systems are judged on several measures that pull against each other [1].
- Latency. How long one request takes, often measured at the 95th or 99th percentile, since slow outliers affect users most.
- Throughput. How many requests per second the system handles.
- Cost. Hardware and energy per prediction.
- Availability. Staying up despite failures and updates.
- Quality. Returning correct, consistent predictions.
Improving one usually costs another. Bigger batches raise throughput but add waiting time. More replicas lower latency under load but cost more.
Anatomy of a serving system
- Interface. REST or gRPC endpoints, or a streaming interface for tokens from a language model; see AI API.
- Gateway and routing. Authentication, rate limits, and routing of requests to model versions.
- Preprocessing. Turning raw input into the format the model expects, matching training; see data preprocessing.
- Inference runtime. The software that runs the model efficiently on hardware; see inference engine.
- Hardware. CPUs, GPUs or other AI chips.
- Postprocessing. Converting raw model outputs into useful results, such as labels or text.
- Model registry. A store of versioned models.
- Monitoring and logging. Metrics and traces; see AI observability.
- Orchestration. Systems such as Kubernetes that start, stop and scale replicas.
Key techniques
### Batching
Running many requests together makes better use of parallel hardware. Dynamic batching waits a few milliseconds to collect requests into a batch. For language models, continuous batching adds new requests to a running batch as others finish, which greatly improves throughput [2].
### Autoscaling
Traffic varies over the day. Autoscaling adds replicas when demand rises and removes them when it falls, using signals such as queue length or GPU use. Scaling GPU services is slower than scaling web servers because large models take time to load, so teams keep warm capacity.
### Caching
Identical or similar requests can reuse earlier answers. Language model servers also cache the attention keys and values of repeated prompt prefixes, saving computation.
### Model optimization
Quantization, pruning, distillation and compilation make models faster and smaller; see model compression. Optimized kernels and runtimes extract more from the hardware.
### Parallelism across devices
Very large models are split over several GPUs using tensor or pipeline parallelism; see distributed computing.
Serving large language models
Language models generate text one token at a time, and each new token depends on all earlier ones. Serving performance is often limited by memory bandwidth and by the key-value cache, which grows with the length of the conversation. Inefficient handling of this cache wastes memory and limits how many requests fit on a GPU. The PagedAttention approach, which manages cache memory in small blocks like an operating system‘s virtual memory, raised throughput substantially in published tests [2]. Other methods include speculative decoding, where a small model drafts tokens that a large model verifies, and prefix caching. Metrics specific to language models include time to first token and tokens per second.
Serving options
- Managed endpoints. Cloud platforms host models for you; see cloud AI.
- Open-source servers. Examples include TensorFlow Serving, TorchServe, NVIDIA Triton, KServe, Ray Serve, vLLM and Hugging Face Text Generation Inference.
- Serverless inference. Functions that scale to zero when idle.
- On-device runtimes. For phones and sensors; see edge AI.
- Hosted model APIs. Providers run the models, and customers pay per request or token.
Reliability and safety
Production serving needs timeouts, retries, circuit breakers and fallbacks, such as a simpler model when the main one is slow. Inputs should be validated, since malformed or hostile inputs can crash a service or manipulate a model. Access control and encryption protect data and the model itself. For generative systems, filters and policies reduce harmful outputs; see AI safety. Teams roll out new versions gradually and keep the ability to roll back.
Capacity planning and cost
To size a serving system, teams measure how many requests per second one replica can handle at an acceptable latency, then divide expected peak traffic by that number and add a safety margin. Load tests that replay realistic traffic reveal bottlenecks before launch. For language models, the unit of work is the token, so planning uses tokens per second and the typical length of prompts and answers, which vary widely between uses.
Cost per request depends on hardware price, how fully the hardware is used and the efficiency of the software. Raising utilization is the main lever: a GPU that is busy most of the time is far cheaper per request than one idling between bursts. Teams therefore combine batching, right-sized models, quantization, request routing to cheaper models for easy queries and scheduling of non-urgent jobs in off-peak hours. Reserved capacity gives lower prices for steady load, while on-demand capacity covers spikes.
Testing and validation before release
Before a new model version reaches users, serving teams run several checks. Unit tests confirm that inputs and outputs have the expected format. Regression tests compare the new model’s answers with a saved set of cases. Load tests check latency and stability under pressure. Shadow traffic tests compare behavior on real requests without affecting users. Only after these checks pass does a canary release begin. Logging every request and response, with privacy safeguards, lets teams reproduce problems and build evaluation sets from real use.
FAQs
What is model serving? It is hosting a trained model behind an interface so applications can send inputs and receive predictions.
What is the difference between model serving and model deployment? Deployment is the whole process of releasing a model to production. Serving is the runtime part that answers requests.
What is dynamic batching? It is grouping several incoming requests together so hardware processes them at once, improving throughput.
Why is serving large language models hard? They are large, generate tokens one at a time, and need lots of memory bandwidth and cache space.
What tools are used for model serving? Examples include TensorFlow Serving, TorchServe, Triton, KServe, Ray Serve and vLLM.
More in AI Data and Infrastructure
Key facts
- Serving systems balance latency, throughput and cost, and dynamic batching groups requests together to use hardware more efficiently1.
- Large language model serving is often limited by memory bandwidth and by the size of the key-value cache that stores past attention states2.
- Techniques such as continuous batching and paged attention increased the number of requests language model servers can handle per GPU2.
Related concepts
Quick checkWhat is model serving?Show answer
Model serving is the practice of hosting a trained machine learning model behind an interface, usually an API, so that applications can send it inputs and quickly receive predictions at scale and with high reliability.
Sources
- Daniel Crankshaw et al., Clipper: A Low-Latency Online Prediction Serving System, USENIX NSDI, https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/crankshaw
- Woosuk Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, Symposium on Operating Systems Principles, https://arxiv.org/abs/2309.06180
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
