Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Small Language Model

nounLarge Language Modelsalso SLM, compact language model, on-device language model

In one line

A small language model is a language model with relatively few parameters, typically from a few hundred million to about ten billion, designed to run efficiently on limited hardware such as phones and laptops.

A small language model, often shortened to SLM, is a language model with relatively few parameters, typically from a few hundred million to around ten billion, that is designed to run efficiently on limited hardware such as phones, laptops and single servers. It does the same basic job as a large language model, reading text and predicting the next token, but at a much smaller scale. The line between small and large is not fixed and moves as hardware improves. A model that counted as large in 2020 would be called small today. SLMs have become a major trend because they offer lower cost, faster responses and the ability to keep data on a user’s device.

History of small language models

Early neural language models were small by today’s standards, with millions of parameters. As the transformer enabled huge models such as GPT-3 with 175 billion parameters, attention focused on scale. From 2022, researchers showed that smaller models trained on more and better data could match larger ones. DeepMind’s Chinchilla study argued for training smaller models longer. In 2023, Meta released Llama models of 7 billion parameters and above, and Microsoft’s Phi-1 and Phi-2 showed that models of 1 to 3 billion parameters trained on high-quality, textbook-like data could do well at coding and reasoning. Google’s Gemma, Mistral 7B, Alibaba’s Qwen, Apple’s on-device models and many others followed, and by 2024 and 2025, phones and laptops shipped with built-in small models.

How small language models are built

SLMs use the same architecture as larger models but make several design and training choices to get the most from limited size.

  • High-quality data: careful filtering and curated or synthetic data let small models learn more from fewer tokens.
  • Longer training: training on trillions of tokens, far beyond what size-optimal rules would suggest, because inference cost matters more than training cost for a widely used model.
  • Distillation: small models learn from a large teacher. See model distillation.
  • Compression: pruning and quantization shrink the model further, for example to 4-bit weights.
  • Efficient architectures: grouped-query attention, shared embeddings and other tricks lower memory and compute needs.
  • Specialization: fine-tuning on a narrow domain such as customer support or medical coding.

Examples

Well-known small models include Microsoft’s Phi family, Google’s Gemma, Meta’s Llama small versions, Mistral’s 7B models, Alibaba’s Qwen small models, Hugging Face’s SmolLM, Apple’s on-device foundation model and several assistants built into phones. Open weights allow developers to run and adapt them locally with tools designed for laptops.

Benefits

  • Cost: serving a small model costs a fraction of a large one per request, and training or fine-tuning is affordable for small teams.
  • Speed and latency: responses begin quickly, which suits real-time and interactive uses.
  • Privacy and control: models running on a device or a private server keep sensitive data in-house.
  • Offline use: they work without an internet connection, important in remote areas and for reliability.
  • Energy efficiency: lower power use per query.
  • Customization: easier to fine-tune and audit for a specific task.

Limitations

Small models generally know fewer facts, handle long and complex reasoning less well, follow intricate instructions less reliably and are more likely to hallucinate on rare topics. Their multilingual ability may be narrower. They have shorter practical context windows on small devices. Safety behavior can be weaker unless carefully trained. For the hardest tasks, large models still lead, so many systems route simple queries to a small model and difficult ones to a large model.

Applications

SLMs power on-device writing suggestions, message summaries, smart replies, voice assistants, offline translation, private note-taking assistants, code completion on laptops, customer support triage, document classification and extraction, game characters and assistants inside cars and appliances. In businesses, specialized SLMs handle repeated narrow tasks at low cost, as parts of agent systems where a planner might use a large model and workers use small ones. See intelligent agents.

Small language models compared with large ones

Large models offer broad knowledge, stronger reasoning and more general ability, but cost more and typically run in the cloud. Small models trade breadth for efficiency and privacy. A good choice depends on the task. Classification, extraction and templated writing often suit small models well, while open-ended research, difficult mathematics and complex coding benefit from larger ones. Retrieval of documents can help small models, because they can look up facts instead of storing them, see vector database.

Deployment tips

Measure the model on your own tasks, check memory and speed on target hardware, use quantized versions where needed, set clear instructions and constraints, add retrieval for facts and keep a fallback to a larger model or a human for hard cases. Check the license of the model and any restrictions on commercial use, and update to newer versions as they appear.

A worked example

A hospital wants a tool to draft discharge summaries without sending patient data to the cloud. It picks an open 3-billion-parameter model, quantizes it to 4 bits so it runs on a standard server, and fine-tunes it on de-identified summaries in the hospital’s style. Doctors review every draft. Compared with a very large cloud model, the small model is somewhat less fluent on rare conditions, so the hospital adds retrieval of its own guidelines and has staff flag mistakes to improve the system.

Why small models can punch above their weight

Several findings explain the progress of small models. Scaling studies showed that many early large models were undertrained, so smaller models given far more data can match them. Data quality matters at least as much as quantity, since text that is clear, accurate and educational teaches a model faster than noisy web pages. Training on the outputs of stronger models gives small models a dense signal about good reasoning, and techniques such as mixture-of-experts, grouped attention and better optimizers add efficiency. As a result, a 3-billion-parameter model today can outperform a 175-billion-parameter model from a few years ago on many benchmarks, though it still lags the best large systems on knowledge-heavy tasks.

Governance and responsible use

Because small models can run anywhere, including offline and outside a provider’s monitoring, organizations should set rules for approved models, track versions, test for safety and bias before deployment and review data handling. Developers should publish model cards that describe training data, intended use and known limits, and users should treat outputs as drafts. On-device deployment helps privacy but also means that updates and fixes must reach many devices.

Future directions

Expect continued gains in quality per parameter, better reasoning in small models, routing systems that combine small and large models, hardware with dedicated neural processors and tools that let people fine-tune models on their own devices. These advances point toward a world in which much everyday AI runs locally, with large models reserved for the hardest problems.

FAQs

How small is a small language model? Usually under about 10 billion parameters, though there is no fixed rule.

Can a small language model run on a phone? Yes. Models of a few billion parameters, often quantized, run on recent phones.

Are small models less safe? Not necessarily, but they need their own safety testing, since they may not inherit the safeguards of larger models.

Key facts

  • Small language models generally have from under one billion to about ten billion parameters, compared with hundreds of billions or more for frontier models.1
  • Models such as Microsoft's Phi series, Google's Gemma and Meta's smaller Llama versions show that carefully trained small models can handle many tasks well.2
  • Small models can run on devices, which improves privacy, cost and latency, but they usually have less knowledge and weaker reasoning than very large models.3

Related concepts

Quick checkWhy use a small language model instead of a large one?Show answer

It is cheaper, faster and can run on a phone, laptop or private server, which helps with cost, latency and privacy, and it often performs well enough on focused tasks.

Sources

  1. Gunasekar et al., Textbooks Are All You Need, https://arxiv.org/abs/2306.11644
  2. Gemma Team, Gemma: Open Models Based on Gemini Research and Technology, https://arxiv.org/abs/2403.08295
  3. Belcak et al., Small Language Models are the Future of Agentic AI, https://arxiv.org/abs/2506.02153

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue