Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Model Compression

nounAI Data and Infrastructurealso neural network compression, network pruning, model optimization

In one line

Model compression is a family of techniques, including pruning, quantization and knowledge distillation, that make neural networks smaller and faster while keeping most of their accuracy.

Model compression is a family of techniques that make a trained neural network smaller and faster while keeping most of its accuracy. Modern models can have billions of parameters, which require lots of memory, energy and time. Compression reduces these costs so that models can run on phones and sensors, respond faster, and be served more cheaply in data centers. It is central to edge AI, model serving and the broader effort to make AI efficient.

Why compress models?

  • Memory. A model with 70 billion parameters stored as 16-bit numbers needs about 140 gigabytes, more than a single GPU holds.
  • Speed. Fewer or cheaper operations mean quicker answers.
  • Cost and energy. Less hardware per request.
  • Deployment. Phones, cars and wearables have tight limits.
  • Privacy. On-device models keep data local.

Main techniques

### Pruning

Pruning removes parts of a network that contribute little. Unstructured pruning zeroes out individual weights, creating sparse matrices. Structured pruning removes whole neurons, channels, attention heads or layers, which gives real speedups on ordinary hardware. A typical process trains the network, prunes the least important parts by a criterion such as small magnitude, and fine-tunes to recover accuracy, sometimes repeating the cycle. The lottery ticket hypothesis suggests that large networks contain small subnetworks, found by pruning, that can be trained to similar accuracy, which helps explain why pruning works [3].

### Quantization

Quantization stores weights, and sometimes activations, using fewer bits, for example 8-bit or 4-bit integers instead of 32-bit floating-point numbers. This reduces memory by four to eight times and speeds arithmetic on supporting hardware. It can be applied after training, which is quick but may lose accuracy, or during training, called quantization-aware training. See quantization.

### Knowledge distillation

In distillation, a small student model is trained to imitate a large teacher model, learning from the teacher’s output probabilities as well as from the original labels. The probabilities carry richer information than hard labels, such as which wrong answers are nearly right [2]. Many compact language and vision models are built this way; see model distillation.

### Low-rank factorization

A large weight matrix can often be approximated by the product of two much smaller matrices. This reduces parameters and computation. Methods such as LoRA use the same idea for efficient fine-tuning; see LoRA and fine-tuning.

### Efficient architectures

Instead of compressing after the fact, designers build efficient networks from the start, such as MobileNet and EfficientNet for images, or smaller transformers. Neural architecture search automates part of this. Techniques such as mixture-of-experts activate only part of a model for each input; see mixture of experts.

### Weight sharing and coding

Groups of weights can share a single stored value, and the values can be entropy coded, as Huffman coding does for text files. The Deep Compression pipeline combined pruning, quantization and coding to shrink image models by 35 to 49 times without loss of accuracy on its tests [1].

Compression for large language models

For language models, the most used methods are 8-bit and 4-bit quantization, which allow models to run on a single consumer GPU or laptop; distillation into smaller models; pruning of layers and heads; compressing the key-value cache used during generation; and speculative decoding, which speeds up generation without changing outputs. Popular formats and tools include GGUF and GPTQ-style quantization, among others, and they change rapidly.

Trade-offs and evaluation

Compression usually costs some accuracy, and the loss is not uniform. A compressed model may match the original on average benchmarks but fail on rare inputs, long reasoning or particular languages. It is therefore important to evaluate on the real task, including subgroups and edge cases; see model evaluation. Theoretical savings, such as fewer parameters, do not always become real speed, which depends on the hardware and software; sparse weights, for instance, only help if the chip can exploit them. Measure latency, memory and energy on the target device.

Workflow

  1. Set targets for size, latency and accuracy.
  2. Start with a strong baseline model.
  3. Try the simplest method first, usually post-training quantization.
  4. Add pruning or distillation if more is needed.
  5. Fine-tune to recover accuracy.
  6. Benchmark on real hardware and test thoroughly.
  7. Deploy and monitor; see model deployment.

A worked example

Suppose a company has an image classifier that takes 120 milliseconds per photo on a phone and uses 90 megabytes, but the product needs under 40 milliseconds and 25 megabytes. The team first applies post-training 8-bit quantization, which cuts the size to about a quarter and speeds inference on the phone’s neural accelerator, with a small accuracy drop. Next they prune a fraction of the channels in the largest layers and fine-tune for a few epochs to win back accuracy. Finally, they distill from the original full-size model into the pruned network, which recovers most of the remaining gap. They test the result on the phone itself, on a held-out set that includes dim and blurry photos, confirm it meets the targets and release it to a small share of users first.

Combining and sequencing methods

The techniques are complementary and are often stacked. A common order is architecture choice, then distillation, then pruning with fine-tuning, then quantization, and finally compilation for the target chip. Each step can interact with the others: heavy pruning changes weight distributions in ways that affect quantization, so quantization-aware training after pruning can help. Teams keep an experiment log of size, speed and accuracy at each stage so that they can identify which step caused any regression.

Limits of compression

There is a floor below which models cannot be shrunk without losing ability, since knowledge stored in parameters is limited by their number and precision. Research on scaling shows that smaller models can be made much better by training on more and cleaner data for longer, so compression and better training work together. For tasks that need broad knowledge or long reasoning, a compressed model may still need to defer to a larger one in the cloud; see cloud AI.

FAQs

What is model compression? It is reducing the size and computing cost of a neural network while keeping most of its accuracy.

What is the difference between pruning and quantization? Pruning removes parts of the network, while quantization stores numbers using fewer bits.

What is knowledge distillation? Training a small student model to imitate a larger teacher model.

Does compression reduce accuracy? Often slightly, but good methods keep most of it, and results should always be tested on the real task.

Can large language models be compressed? Yes, mainly through quantization, distillation and pruning, allowing them to run on smaller hardware.

More in AI Data and Infrastructure

Key facts

  • The Deep Compression method combined pruning, quantization and Huffman coding to shrink networks by 35 to 49 times on standard image models with no loss of accuracy1.
  • Knowledge distillation trains a small student model to imitate the output probabilities of a larger teacher model2.
  • The lottery ticket hypothesis proposed that large networks contain small subnetworks that can be trained to similar accuracy3.

Related concepts

Quick checkWhat is model compression?Show answer

Model compression is a family of techniques, such as pruning, quantization and knowledge distillation, that reduce the size and computing cost of a neural network while keeping most of its accuracy, so it can run faster, cheaper and on smaller devices.

Sources

  1. Song Han, Huizi Mao and William Dally, Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding, https://arxiv.org/abs/1510.00149
  2. Geoffrey Hinton, Oriol Vinyals and Jeff Dean, Distilling the Knowledge in a Neural Network, https://arxiv.org/abs/1503.02531
  3. Jonathan Frankle and Michael Carbin, The Lottery Ticket Hypothesis, International Conference on Learning Representations, https://arxiv.org/abs/1803.03635

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue