Tokenization is the process of splitting text into smaller units called tokens and converting them to numbers so that a language model can process them. A tokenizer takes a sentence such as Unbelievable results! and returns a list of pieces, for example un, believ, able, results and an exclamation mark, each with an integer ID. The model never sees raw letters. It sees sequences of token IDs, looks up a learned embedding for each and predicts the next token. Tokenization is the first step of every natural language processing pipeline and is a hidden but important design choice in large language models.
History of tokenization
Early text processing split words at spaces and punctuation, lowercased everything and removed rare words, often replacing them with an unknown token. Linguists developed more careful rules for languages without spaces, such as Chinese and Japanese, and for languages with rich word forms. For neural models in the 2010s, large word vocabularies were costly and could not handle new words. In 2015, Rico Sennrich and colleagues adapted the byte pair encoding compression algorithm to build vocabularies of subword units for machine translation. Google introduced WordPiece for speech and then BERT, Taku Kudo released SentencePiece in 2018 and GPT-2 in 2019 used byte-level BPE, which can encode any text. Since then, tokenizers have grown larger, with vocabularies of 100,000 to over 250,000 entries in multilingual models.
Levels of tokenization
- Word-level: each word is a token. It gives short sequences and simple meaning, but the vocabulary is huge and new words are unknown.
- Character-level: each character is a token. The vocabulary is tiny and nothing is unknown, but sequences are long and meaning is spread out.
- Subword-level: common words are single tokens and rare words are built from pieces. This is the standard approach for modern models.
- Byte-level: the text is treated as raw bytes, which guarantees coverage of any language, symbol or emoji.
Main subword algorithms
### Byte pair encoding
BPE starts with single characters or bytes and repeatedly merges the most frequent adjacent pair in a training corpus, adding the merged symbol to the vocabulary, until a target size is reached. To tokenize new text, the learned merges are applied in order. It is simple and fast and is used in GPT models.
### WordPiece
WordPiece also builds a vocabulary by merging, but chooses merges that most increase the likelihood of the training data under a language model. It marks word-internal pieces with a prefix and is used in BERT.
### Unigram language model
The unigram method starts with a large set of candidate pieces and removes those that least affect the likelihood of the corpus, ending with a vocabulary of a desired size. It can produce several possible segmentations with probabilities, which can be sampled during training as regularization.
### SentencePiece
SentencePiece is a library that treats text as a stream of characters including spaces, so it needs no language-specific pre-splitting and works for many languages. It supports BPE and unigram models.
The tokenization pipeline
- Normalization: clean the text, for example by applying Unicode normalization and handling case and accents.
- Pre-tokenization: split roughly at spaces, punctuation or digits.
- Model: apply the subword algorithm to produce pieces.
- Post-processing: add special tokens for sequence start, end and separation, and map pieces to IDs.
Decoding reverses the steps to turn token IDs back into text. The tokenizer must match the model exactly, since a model trained with one vocabulary will produce nonsense with another.
Effects on model behavior
Tokenization shapes what models find easy. Because words are chunks, models may struggle with spelling tasks, character counting and reversing strings. Numbers can be split inconsistently, which affects arithmetic, so some tokenizers split digits individually. Code with unusual whitespace can use many tokens. Names from underrepresented languages may split into many pieces, reducing quality. Spaces and capitalization can change tokenization and so slightly change outputs. Rare vocabulary entries that appeared little in training, called glitch tokens, can produce erratic behavior.
Tokenization and fairness
Tokenizers trained mostly on English text split English efficiently but other languages into longer sequences. Studies have found that the same content can need several times more tokens in languages such as Shan or Burmese than in English, which raises cost, reduces available context and can lower accuracy for those users. Developers respond with more balanced training data for tokenizers, larger vocabularies and research into tokenizer-free models.
Beyond text
Images are divided into patches, audio is compressed into discrete codes and video into spacetime patches, which act as tokens for a transformer. See multimodal AI and token.
Practical tips
Use the tokenizer that matches your model, count tokens before sending long inputs, avoid splitting numbers and identifiers unnecessarily, test prompts in all target languages and be careful when modifying vocabularies, since changes require retraining or careful adaptation of the model.
A worked example
A small BPE vocabulary learns that the pair of letters t and h is common and merges them into th, then merges th and e into the. After thousands of merges, the word the becomes one token and a rare word such as tokenizer may become the pieces token and izer. When a developer sends the text to a model, the tokenizer produces IDs such as 1169 and 7345, which the model converts to embeddings. The reply is generated as IDs and decoded back into words.
Tokenization in other fields
The word tokenization has other meanings. In data security, it means replacing a sensitive value, such as a payment card number, with a random stand-in that can be mapped back only by a secure system. In blockchain, it refers to representing an asset as a digital token. This article concerns the language-processing sense, where tokens are units of text. Keeping these meanings apart avoids confusion when reading about AI and data protection together.
Training a tokenizer
To build a tokenizer, developers choose a representative text sample, a target vocabulary size and an algorithm, then run the training procedure to learn merges or pieces. Vocabulary size is a trade-off. A larger vocabulary shortens sequences and reduces cost per document but enlarges the embedding table and makes rare tokens less well trained, while a smaller one lengthens sequences. Teams check coverage of their languages and domains, test on code and numbers and inspect the most common and rarest tokens for surprises such as stray web fragments. Once the model is trained, the tokenizer is frozen and shipped with it.
Future directions
Researchers are exploring models that learn how to segment text during training, byte-level architectures that group bytes into patches dynamically and vocabularies designed for fair treatment of many languages. If these approaches succeed, tokenization may become less of a fixed preprocessing step and more of a learned part of the model.
FAQs
Is tokenization the same as embedding? No. Tokenization splits text into units and assigns IDs, while embedding maps each ID to a vector of numbers that the model uses.
Can I change a model’s tokenizer? Not without retraining or careful adaptation, because the model learned its weights for a specific vocabulary.
Why do models make spelling mistakes? They see tokens as chunks, so individual letters are not directly visible.
More in Large Language Models
Key facts
- Byte pair encoding, a compression method from 1994, was adapted for neural machine translation in 2015 to build subword vocabularies.1
- SentencePiece, released in 2018, tokenizes raw text without language-specific preprocessing and supports BPE and unigram models.2
- Tokenizer choices affect cost, context length and fairness across languages.3
Related concepts
Quick checkWhy do language models use subword tokenization?Show answer
It keeps the vocabulary manageable while still representing any word, since rare or new words are built from smaller known pieces instead of becoming unknown.
Sources
- Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units, https://arxiv.org/abs/1508.07909
- Kudo and Richardson, SentencePiece: A simple and language independent subword tokenizer, https://arxiv.org/abs/1808.06226
- Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages, https://arxiv.org/abs/2305.15425
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
