In a language model, a token is a basic unit of text that the model reads and generates. A token may be a whole word, such as house, part of a word, such as the pieces un and believ and able in unbelievable, a punctuation mark, a space plus a word or even a single byte. Neural networks work with numbers, not letters, so text is first split into tokens and each token is replaced by an integer ID from a fixed vocabulary. The model then predicts one token at a time. Tokens are the currency of large language models, since the length of a prompt, the size of the context window and the price of using a model are all measured in tokens.
Why models use tokens
A model could work with individual characters, which gives a tiny vocabulary but very long sequences, or with whole words, which gives short sequences but a huge vocabulary and no way to handle new or misspelled words. Subword tokens strike a balance. Common words get a single token, while rare words are broken into familiar pieces, so the model can handle any text with a vocabulary of tens of thousands to a few hundred thousand entries. This keeps sequences reasonably short and avoids unknown words.
How tokens are created
The process is called tokenization, and the program that does it is a tokenizer. Popular methods include the following.
- Byte pair encoding (BPE): starts with characters or bytes and repeatedly merges the most frequent adjacent pairs in a large text sample until a target vocabulary size is reached. Used in GPT models.
- WordPiece: similar to BPE but chooses merges that most improve the likelihood of the training data. Used in BERT.
- Unigram language model: starts with a large set of candidate pieces and removes those that matter least. Used in SentencePiece.
- Byte-level tokenization: works on raw bytes, so any text, including emoji and unusual scripts, can be represented.
The tokenizer is trained before the model and then fixed. Each token has an ID and a learned embedding, a vector that represents its meaning in context.
How many tokens is a text?
In English, a token averages about four characters, so 100 tokens is roughly 75 words, and a page of text is about 500 tokens. The count varies by language. Languages with different scripts or less representation in the tokenizer’s training data, such as Burmese, Amharic or Tamil, can need several times more tokens for the same meaning, which makes use more expensive and shrinks the effective context. Numbers, code, rare names and emoji can also split into many tokens. Tools such as online tokenizer pages and libraries, for example tiktoken, show exactly how a text is split.
Tokens, cost and limits
Providers usually bill separately for input tokens, the prompt, and output tokens, the reply, and often charge more for output. Each model has a maximum number of tokens it can handle in one request and a maximum for the answer. Speed is also measured in tokens per second. Efficient prompts, short outputs and caching of repeated prefixes lower cost. When estimating cost, remember that conversation history is sent again with each turn in many applications, so long chats use more tokens over time.
Special tokens
Tokenizers include special tokens that carry structure rather than text. They mark the beginning and end of a sequence, separate user and assistant turns in a chat, pad shorter sequences in a batch, mask words in training or indicate the start of an image or tool call. Chat models are trained with specific special tokens, so developers should use the template the model expects.
Tokens beyond text
The idea extends beyond words. Image models split a picture into patches that act as tokens, audio models use discrete tokens from neural codecs and video models use spacetime patches. This lets one transformer architecture process text, images, sound and video in a similar way. See multimodal AI.
Quirks caused by tokenization
Many odd behaviors of language models trace back to tokens. Models may struggle to count letters in a word, reverse a string or do arithmetic on long numbers because they see chunks and not letters or digits. Uncommon names may be split in ways that hurt recall. Extra spaces or different capitalization can change token splits and slightly change answers. Glitch tokens, rare strings in the vocabulary that were barely seen in training, have caused strange outputs in some models.
Fairness and design issues
Because tokenizers are trained mostly on certain languages, speakers of other languages can pay more and receive less usable context for the same content. Researchers and developers are working on more balanced vocabularies and on tokenizer-free models that work directly on bytes. Choices about tokens also affect security, since attackers may use unusual tokens to try to bypass filters.
Tips for working with tokens
Count tokens with the provider’s tool before sending long inputs. Remove unnecessary boilerplate, ask for concise answers when appropriate, set a maximum output length and keep a record of usage. For multilingual products, test token counts in each language and budget accordingly.
A worked example
Suppose a tokenizer splits the sentence Tokenization is unbelievably useful into the pieces Token, ization, is, un, believ, ably, useful. The model sees seven token IDs. If a developer sends this sentence with a prompt of 400 other tokens and expects a 200-token reply, the request uses about 607 tokens in total, which fits easily within a 4,000-token window and is billed accordingly.
Token probabilities and sampling
At each step, a model assigns a probability to every token in its vocabulary, often more than 100,000 of them, and a sampling rule picks one. Settings such as temperature, top-k sampling and top-p sampling change how that choice is made. Some services let developers see the probabilities of the most likely tokens, called log probabilities, which is useful for judging how confident the model was and for classification tasks, where a single token such as yes or no can be the answer.
History of tokens in NLP
Early systems treated each word as an atom and used a fixed vocabulary, replacing unknown words with a special unknown token, which lost information. In 2015 and 2016, researchers applied byte pair encoding, a compression algorithm from 1994, to neural machine translation, showing that subword units handled rare words well. SentencePiece in 2018 made tokenization independent of language-specific rules, and byte-level BPE in GPT-2 ensured that any string could be encoded. Recent work explores larger vocabularies for multilingual use and byte-level models that avoid a tokenizer.
Future directions
Researchers are testing tokenizers that treat languages more equally, models that learn their own segmentation, and architectures that operate on bytes or patches while remaining efficient. Whatever the method, the idea of breaking input into units that a network can predict one by one is likely to remain central to language and multimodal AI.
Related concepts
Tokens are produced by tokenization and are the basic input to every language model, so understanding them explains many practical limits of language AI.
FAQs
Is a token the same as a word? No. A token can be a word, part of a word or punctuation, and many common English words are one token.
Why do other languages use more tokens? Tokenizers are usually trained mostly on English and a few major languages, so text in less represented languages splits into more pieces.
How can I count tokens? Use the model provider’s tokenizer tool or a library such as tiktoken.
More in Large Language Models
Key facts
- A token is typically a word, a piece of a word or a punctuation mark, and in English one token averages about four characters or three quarters of a word.1
- Most modern models use subword methods such as byte pair encoding to build their token vocabulary.2
- Model limits and prices are usually quoted in tokens, and the same text can use different numbers of tokens in different languages.3
Related concepts
Quick checkWhat is a token in a language model?Show answer
It is the unit of text the model processes, usually a word, part of a word or symbol, converted to a number so the neural network can work with it.
Sources
- OpenAI, What are tokens and how to count them, https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them
- Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units, https://arxiv.org/abs/1508.07909
- Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages, https://arxiv.org/abs/2305.15425
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
