Top-k sampling is a way of choosing the next token when a language model generates text. At each step, the model assigns a probability to every token in its vocabulary. Instead of sampling from all of them, which risks picking very unlikely and often nonsensical tokens, top-k sampling keeps only the k most probable candidates, rescales their probabilities so that they add up to one and picks one at random in proportion to those probabilities. The method keeps the variety of random sampling while cutting off the long tail of poor choices. It is one of the standard controls in text generation and is often used together with temperature and top-p sampling.
History of top-k sampling
Early neural text generators used greedy decoding, which always picks the top token, or beam search, which tracks several best sequences. These are good for translation but tend to give repetitive or bland text in open-ended writing. Pure random sampling from the full distribution produced more varied text but often included strange words, since a vocabulary of tens of thousands of tokens has a long tail whose combined probability is not small. In 2018, Angela Fan, Mike Lewis and Yann Dauphin used top-k sampling with k equal to 10 for story generation and showed that it produced more interesting text than beam search. OpenAI’s GPT-2 release in 2019 popularized k equal to 40 in demonstration samples. In the same year, Holtzman and colleagues introduced top-p sampling as an adaptive variant.
How top-k sampling works
- Compute probabilities: the language model produces logits for all tokens, and temperature may be applied, then a softmax gives probabilities.
- Rank and truncate: sort tokens by probability and keep the top k. All others are discarded by setting their probability to zero.
- Renormalize: divide the remaining probabilities by their sum so they total one.
- Sample: draw one token from this reduced distribution.
- Repeat: append the token and continue with the next step.
If k equals 1, the method reduces to greedy decoding. If k equals the size of the vocabulary, it becomes ordinary random sampling.
Example
Suppose the model must continue The cat sat on the, with probabilities mat 40 percent, sofa 25 percent, floor 15 percent, roof 8 percent, moon 5 percent and many other words sharing the remaining 7 percent. With k equal to 3, only mat, sofa and floor remain, and their probabilities are rescaled to about 50, 31 and 19 percent. The model then chooses among those three. The words roof and moon, and the rare tokens, cannot be chosen at that step.
Choosing k
Small values of k, such as 1 to 10, give focused and predictable text. Values around 40 to 50 are common defaults and give a good balance. Very large values approach unrestricted sampling. The best k depends on the task, the model and the other settings. For tasks needing accuracy, such as answering factual questions or writing code, a small k or a low temperature is wiser, and for creative writing, a larger k allows more variety.
Limitations
The main weakness is that k is fixed, while the number of reasonable next tokens changes from step to step. When the model is very sure, such as after the first part of a famous phrase, only one or two tokens make sense, yet k equal to 40 still allows 38 poor options with some chance of being picked. When the model is uncertain, such as at the start of a sentence, many tokens are reasonable, yet k equal to 40 may cut off good ones. This is the reason for adaptive methods such as top-p, which adjust the number of candidates to the shape of the distribution, and for newer methods such as min-p sampling, which keeps tokens whose probability is above a fraction of the top token’s.
Top-k compared with other decoding methods
- Greedy decoding: always picks the single most likely token, which is deterministic but may be repetitive.
- Beam search: keeps several candidate sequences and picks the best, suited to translation and summarization.
- Temperature sampling: reshapes the whole distribution but does not remove the tail.
- Top-p sampling: keeps the smallest set of tokens whose probabilities add up to p, adapting its size to the model’s confidence.
- Top-k sampling: keeps a fixed number of tokens.
In practice, systems often combine temperature with top-k or top-p, applying temperature first and then truncation.
Applications
Top-k sampling is used in chatbots, story and poem generation, dialogue systems, code completion with variety, and in generative models for music and images that predict tokens one at a time. It is available as a setting in most model libraries and in many application programming interfaces, although some providers expose only temperature and top-p.
Tips for use
Change one setting at a time, test on real examples and keep a log. If results are repetitive, increase k or temperature or add a repetition penalty. If results are erratic, lower k or temperature. For reproducible output, use a fixed random seed and the same settings. Remember that good prompts and a good model matter more than fine sampling choices.
A worked example
A developer builds a bedtime-story app. With greedy decoding, the stories begin the same way every night. With unrestricted sampling at temperature 1, some stories include odd or ungrammatical words. She sets k equal to 40 and temperature to 0.8, which gives varied stories that remain readable. For the parent-facing summaries, where accuracy matters, she uses a temperature of 0.2 and k equal to 10.
Implementation notes
In code, top-k sampling is a few lines. A library sorts the logits, finds the value of the kth largest and masks every logit below it with minus infinity, so that the softmax gives those tokens zero probability. Frameworks such as Hugging Face Transformers expose it as a top_k argument, and many serving systems apply it on the graphics card for speed. Ties at the kth value can keep slightly more than k tokens in some implementations. When combined with other filters, the order matters. A common order is temperature, then top-k, then top-p, and the result can differ if the order changes.
Quality, diversity and evaluation
Researchers compare decoding methods on measures of fluency, such as perplexity under a reference model, and of diversity, such as the number of distinct n-grams and the repetition rate, together with human judgments of coherence. Studies find a trade-off, as stricter truncation raises coherence while reducing diversity. Because the right balance depends on the task, evaluation should use samples from the real application rather than generic benchmarks.
Future directions
Work continues on sampling methods that adapt to the model’s uncertainty, such as min-p and typical sampling, on decoding that favors factual accuracy and on techniques that let users specify a desired level of diversity directly. Whatever the method, the aim is the same, to choose words that are both fitting and varied.
FAQs
What is the difference between top-k and top-p? Top-k keeps a fixed number of tokens, while top-p keeps as many tokens as are needed to reach a total probability p.
What happens when k is 1? The model always picks the most likely token, which is greedy decoding.
Can I use top-k with temperature? Yes, they are commonly combined, with temperature reshaping the probabilities and top-k trimming the tail.
More in Large Language Models
Key facts
Related concepts
Quick checkWhat does top-k sampling do?Show answer
It keeps only the k most likely next tokens, rescales their probabilities to sum to one and samples one of them, which avoids very unlikely words while keeping variety.
Sources
- Fan, Lewis and Dauphin, Hierarchical Neural Story Generation, https://arxiv.org/abs/1805.04833
- Holtzman et al., The Curious Case of Neural Text Degeneration, https://arxiv.org/abs/1904.09751
- Jurafsky and Martin, Speech and Language Processing, chapter on large language models, https://web.stanford.edu/~jurafsky/slp3/
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
