Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Quantization (Machine Learning)

nounLarge Language Modelsalso model quantization, neural network quantization, low-bit quantization

In one line

Quantization is a model compression technique that stores a neural network's numbers with fewer bits, such as 8 or 4 instead of 16 or 32, to cut memory use and speed up inference.

Quantization is a model compression technique that stores the numbers in a neural network with fewer bits. A trained model usually keeps its weights as 32-bit or 16-bit floating-point numbers. Quantization maps them to lower-precision formats, such as 8-bit or 4-bit integers, which take a half, a quarter or an eighth of the memory. The smaller model loads faster, needs less memory bandwidth, uses less energy and often runs faster on suitable hardware, usually with only a small loss in accuracy. Quantization is a key reason why large language models with billions of parameters can run on a single graphics card, a laptop or a phone.

History of quantization

Reducing numeric precision is an old idea in signal processing and embedded computing. In neural networks, work in the 1990s explored low-precision weights, and in 2015 and 2016, researchers such as Song Han showed with Deep Compression that pruning, quantization and coding could shrink networks by an order of magnitude. BinaryConnect and XNOR-Net pushed to one-bit weights. In 2017 and 2018, Google’s work on integer-only inference made 8-bit quantization practical on mobile chips, and TensorFlow Lite and PyTorch added support. As language models grew, researchers found that large transformer models have outlier values that make simple quantization difficult. LLM.int8 in 2022, GPTQ in 2022, SmoothQuant and AWQ in 2023 and QLoRA in 2023 enabled accurate 8-bit and 4-bit models, and formats such as GGUF made quantized models easy to run locally.

How quantization works

A floating-point number can represent a wide range of values with fine detail. An integer with fewer bits can represent only a limited set of levels. Quantization maps a range of real values onto these levels using a scale, and sometimes a zero point.

  1. Choose the range: find the minimum and maximum, or a clipped range, of the values in a tensor or group of values.
  2. Compute the scale: divide the range by the number of available levels, for instance 255 for 8 bits.
  3. Round: divide each value by the scale and round to the nearest integer.
  4. Store and use: keep the integers and the scale. At run time, the hardware either computes directly in integers or converts the numbers back to approximate real values, a step called dequantization.

The difference between the original and the quantized value is the quantization error. The goal is to keep that error from harming the model’s output.

Types of quantization

  • Post-training quantization (PTQ): converts an already trained model without retraining, often using a small calibration dataset to choose scales. It is fast and cheap, and methods such as GPTQ and AWQ adjust rounding to minimize error.
  • Quantization-aware training (QAT): simulates low precision during training or fine-tuning so the model learns to cope with it. It usually gives better accuracy at very low bit widths but costs more.
  • Weight-only quantization: stores weights in low precision but computes activations in higher precision. Common for language models, where memory bandwidth is the bottleneck.
  • Weight and activation quantization: quantizes both, enabling fast integer arithmetic.
  • Uniform and non-uniform: uniform quantization uses evenly spaced levels, while non-uniform schemes, such as the 4-bit NormalFloat in QLoRA, place levels where values are most common.
  • Per-tensor, per-channel and group-wise: using separate scales for smaller groups of values reduces error.
  • Mixed precision: keeps sensitive layers in higher precision.

Benefits

Quantization cuts the memory needed to store and run a model. A 7-billion-parameter model needs about 14 gigabytes in 16-bit form but only about 4 gigabytes in 4-bit form, which lets it fit on a consumer graphics card or laptop. Moving less data speeds up generation, which is often limited by memory bandwidth, and reduces energy use. It also makes it practical to run models on phones and other edge devices without sending data to the cloud and lowers serving costs for companies.

Trade-offs and limitations

  • Accuracy loss: lower precision introduces error. Eight-bit quantization is usually nearly lossless, 4-bit is often acceptable, and 2-bit or 3-bit can noticeably harm quality.
  • Outliers: a few very large values in transformers can dominate the scale, and handling them takes special techniques.
  • Hardware support: speed gains depend on the chip supporting the chosen format.
  • Task sensitivity: reasoning, mathematics, long context and low-resource languages may degrade more than simple chat.
  • Evaluation burden: a quantized model must be re-tested, since benchmark averages can hide failures.
  • Safety: some studies suggest that compression can change safety behavior, so it should be tested too.

Related compression techniques

Quantization is often combined with other methods. Pruning removes unimportant weights or whole structures. Knowledge distillation trains a small student model to mimic a larger teacher. Low-rank factorization approximates large matrices with smaller ones, and sparse mixture-of-experts designs activate only part of a model for each token. Together, these make language models cheaper to run.

Quantization during training and fine-tuning

Training at lower precision, such as 16-bit brain floating-point or 8-bit floating-point formats, speeds up LLM training and reduces memory use. QLoRA keeps the base model frozen in 4-bit form and trains small LoRA adapters in higher precision, which allows fine-tuning of large models on a single graphics card. Quantization is also used for embeddings in a vector database, where compressing vectors saves storage and speeds search.

Applications

Quantized models run voice assistants on phones, translation on devices without connectivity, on-device image enhancement, local chat assistants on laptops, large-scale cloud inference at lower cost, and models in cars, cameras and sensors. Open-source communities share quantized versions of popular models in formats suited to common tools.

Tips for use

Start with 8-bit or 4-bit weight-only quantization from a trusted source, compare quality with the full-precision model on your own tasks, check speed and memory on your hardware, keep higher precision for sensitive layers if needed and verify the license of the model and quantized variant. Measure before and after, and do not rely on size alone.

A worked example

A small clinic wants a private assistant that summarizes appointment notes on an office computer, without sending data to the cloud. It selects an open 8-billion-parameter language model whose 16-bit version needs about 16 gigabytes of memory, more than the computer has. A 4-bit version needs about 5 gigabytes and fits. Staff test it on 100 de-identified notes, compare summaries with the full model, find them close and deploy it with clinician review of every summary.

FAQs

Does quantization make a model less accurate? Slightly, in most cases. Eight-bit models are usually very close to the original, and 4-bit models are often good enough, but very low bit widths can hurt.

What is the difference between quantization and pruning? Quantization reduces the precision of each number, while pruning removes parameters.

Can I quantize any model? Most neural networks can be quantized, but results vary, and some architectures and tasks are more sensitive than others.

Key facts

  • Quantization represents weights and sometimes activations with lower-precision numbers, such as 8-bit or 4-bit integers instead of 16-bit or 32-bit floating point.1
  • Post-training methods such as GPTQ and AWQ let large language models run in 4 bits with small accuracy loss.2
  • QLoRA combines 4-bit quantization with low-rank adapters to fine-tune large models on a single graphics card.3

Related concepts

Quick checkWhy quantize a machine learning model?Show answer

To reduce its memory footprint, energy use and latency, so a large model can run on cheaper hardware or on a phone, usually with only a small loss of accuracy.

Sources

  1. Jacob et al., Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, https://arxiv.org/abs/1712.05877
  2. Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, https://arxiv.org/abs/2210.17323
  3. Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, https://arxiv.org/abs/2305.14314

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue