Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Language Model

nounLarge Language Modelsalso statistical language model, neural language model, LM

In one line

A language model is a statistical or neural model that assigns probabilities to sequences of words or tokens and is used to predict, generate and understand text.

A language model is a model that assigns probabilities to sequences of words or tokens. Given the words the cat sat on the, it estimates how likely each possible next word is, so that mat is more probable than refrigerator. This simple ability lets a language model score the fluency of a sentence, choose between alternative readings in speech recognition, complete text, translate, summarize and, when run repeatedly, generate whole passages. Language models are the core technology behind modern text generation and the chat assistants known as large language models, and they are a central tool in natural language processing.

History of language models

The roots of language modeling lie in information theory. In 1948 and 1951, Claude Shannon measured how predictable English letters and words are, estimating the entropy of printed English. In the 1980s and 1990s, n-gram models, which estimate the next word from the previous n minus one words, became the standard in speech recognition and machine translation, using counts from large text collections and smoothing methods to handle unseen sequences. In 2003, Yoshua Bengio and colleagues introduced a neural probabilistic language model that learned word embeddings. Recurrent networks and long short-term memory models improved on n-grams in the 2010s. The transformer of 2017 and its use for pretraining, as in BERT and GPT in 2018, led to large language models with billions of parameters, including GPT-3 in 2020 and later systems that power chat assistants.

How language models work

A language model estimates the probability of a sequence by multiplying the probability of each token given those before it. Different model types estimate these probabilities in different ways.

  • N-gram models: count how often short word sequences occur in a corpus and use these counts as probabilities. They are simple and fast, but cannot see beyond a few words and struggle with rare sequences.
  • Neural network language models: represent words as vectors called embeddings and use a network to predict the next word. See embedding.
  • Recurrent models: read text one token at a time and keep a hidden state that summarizes the past. See recurrent neural network.
  • Transformer models: use self-attention to relate every token to every other token in a context window, which captures long-range dependencies and trains efficiently on parallel hardware.

Text is first split into tokens, which may be whole words, parts of words or characters. See tokenization. Training adjusts the model’s parameters, typically by gradient descent, to make the real next token more probable. Quality is measured by perplexity, which indicates how surprised the model is by held-out text, where lower values are better.

Types of language models

  • Autoregressive or causal models: predict the next token from the left context, as in GPT. They are natural for generation.
  • Masked language models: predict hidden words using context from both sides, as in BERT, and are strong for understanding tasks such as classification.
  • Encoder-decoder models: read an input sequence and generate an output sequence, as in T5, for translation and summarization.
  • Small and large models: small language models run on phones and laptops, while large ones have billions or trillions of parameters.
  • Multilingual and multimodal models: handle many languages or combine text with images and audio. See multimodal AI.

Applications

Language models power predictive keyboards and autocomplete, spelling and grammar checkers, machine translation, speech recognition and search. They write and summarize text, answer questions, extract information from documents and generate code. When combined with retrieval, they answer questions using external sources. In science, language models trained on protein sequences or chemical strings help design molecules. In businesses, they draft emails, support customers and analyze feedback.

Strengths and limitations

Language models learn grammar, facts and patterns of reasoning from text and can be adapted to many tasks with little extra data. However, they may generate fluent but false statements, known as hallucinations, because they model likely text and not verified truth. They inherit biases from training data, may reveal memorized private information, can be sensitive to the wording of a prompt and have a limited context window that restricts how much text they consider at once. They also require substantial computing and energy for training.

Ethical and social issues

Questions include copyright in training data, the spread of misinformation and spam, effects on jobs in writing and programming, academic honesty, privacy and fairness across languages and dialects. Developers use filtering, human feedback training, safety testing and documentation such as model cards, and regulators are developing rules on transparency and risk.

A worked example

Suppose a small bigram model has seen the phrase the cat sat many times and the phrase the cat ran fewer times. Given the cat, it assigns higher probability to sat. A transformer model does the same but uses the entire context, so given a story about a hungry cat in a kitchen, it may predict ate or meowed. When the model is used to write text, the system samples from these probabilities, and settings such as temperature change how adventurous the choices are.

Language models compared with search engines and rule-based systems

A search engine retrieves existing documents that match a query, while a language model generates new text from patterns it learned. A rule-based system follows hand-written instructions and is predictable but brittle, whereas a language model generalizes to wording it has never seen. Each has a place. Retrieval keeps answers tied to sources, rules guarantee certain behavior and language models provide flexibility, so modern products often combine all three, for example by letting a model draft an answer from retrieved documents and then applying rules to check the format and block unsafe content.

Evaluating language models

Perplexity measures prediction quality on held-out text, but it does not capture usefulness. Researchers also use benchmarks of reading comprehension, reasoning, knowledge and coding, along with tests of truthfulness, bias and safety. Human raters compare responses for helpfulness and accuracy. Because benchmark questions can leak into training data, careful evaluators create new tests and measure performance on the exact tasks a product needs.

Future directions

Research aims at models that use context more efficiently, are more truthful and can cite sources, adapt to a user’s data without exposing it, run on small devices and learn from fewer examples. Combining language models with tools, memory and planning leads toward intelligent agents that act as well as write, which makes careful evaluation and oversight more important.

FAQs

Is a language model the same as ChatGPT? ChatGPT is a product built on a large language model that was further trained to follow instructions and chat.

What is perplexity? It is a measure of how well a model predicts a text, with lower perplexity meaning better prediction.

Do language models understand language? They capture statistical patterns that support many language tasks, and whether this amounts to understanding is debated.

Key facts

  • A language model assigns a probability to a sequence of words and can predict the next word given the preceding ones.1
  • Claude Shannon studied the statistics of English text in 1948 and 1951, an early step toward statistical language modeling.2
  • Large neural language models based on the transformer, such as GPT, are trained on huge text collections and can be adapted to many tasks.3

Related concepts

In the news

Quick checkWhat does a language model do?Show answer

It estimates how likely different words or tokens are to come next given the text so far, which lets it score, predict and generate language.

Sources

  1. Jurafsky and Martin, Speech and Language Processing, chapters on n-gram and neural language models, https://web.stanford.edu/~jurafsky/slp3/
  2. Shannon, Prediction and Entropy of Printed English, https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf
  3. Vaswani et al., Attention Is All You Need, https://arxiv.org/abs/1706.03762

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue