Pretraining is the first and most expensive stage in building many modern AI models. In this stage, a model learns general patterns from a very large collection of mostly unlabeled data, such as text, images, audio or code, using a self-supervised task that needs no human labels. A language model, for example, reads trillions of tokens and learns to predict the next one. The result is a general-purpose model that can later be adapted to particular tasks by fine-tuning, instruction tuning or prompting. Pretraining is the foundation of large language models and foundation models.
History of pretraining
The idea builds on transfer learning, where knowledge from one task helps with another. In computer vision, researchers around 2012 to 2014 found that networks trained on the labeled ImageNet dataset produced features useful for many other tasks, so pretraining on ImageNet became standard. In language, word embeddings such as word2vec and GloVe, learned from unlabeled text in 2013 and 2014, were an early form of pretraining. In 2018, ELMo, ULMFiT, GPT and BERT showed that pretraining a whole network on unlabeled text and then fine-tuning it greatly improved many language benchmarks. Later models scaled up pretraining dramatically, as with GPT-3 in 2020, and the approach spread to images with methods such as SimCLR and masked autoencoders, to speech with wav2vec 2.0, to multimodal learning with CLIP and to proteins and genomes.
How pretraining works
Pretraining uses self-supervised learning, in which the data provides its own training signal. Common objectives include the following.
- Next-token prediction: the model predicts the next token from the previous ones, as in GPT. This single objective teaches grammar, facts and reasoning patterns, and it is natural for generation. See language model.
- Masked prediction: parts of the input are hidden and the model reconstructs them using context from both sides, as in BERT and masked image models.
- Denoising: the model restores corrupted text or images, as in T5 and diffusion models.
- Contrastive learning: the model learns to place related items, such as an image and its caption, close together and unrelated ones far apart, as in CLIP.
The model, usually a transformer, processes batches of data, computes a loss that measures its prediction error and updates parameters through backpropagation and gradient descent. Training runs over trillions of tokens, takes weeks on thousands of accelerators and produces a checkpoint of learned weights.
What is learned during pretraining
Pretraining captures syntax, vocabulary, common-sense and world knowledge, patterns of reasoning, programming conventions and relationships between concepts, stored in the model’s parameters. It also captures the biases, errors and styles in the data. A pretrained model alone is not a helpful assistant, since it simply continues text in the manner of its training data, so it may answer a question with another question. Later tuning shapes behavior.
Pretraining data
Data sources include web crawls such as Common Crawl, books, encyclopedias, news, scientific articles, code repositories and, for multimodal models, images with captions and video. Preparation involves deduplication, filtering of low-quality or toxic content, language identification, removal of personal information and balancing the mix of domains. Because data quality strongly affects results, developers often weight high-quality sources more heavily. Questions of copyright, consent and privacy surround data collection, and some developers use licensed or synthetic data to address concerns.
Benefits
- Reduces labeled data needs: adapting a pretrained model needs far fewer labeled examples than training from scratch.
- Improves performance: pretrained models reach higher accuracy on many tasks.
- Reuse: one expensive pretraining run supports many applications.
- Few-shot ability: large pretrained models can perform new tasks from instructions or examples.
Limitations and costs
Pretraining at scale needs huge computing power, energy and engineering effort, so only a few organizations can train frontier models. The data may contain errors, bias, private information and copyrighted material. Knowledge is frozen at the training cutoff, so models lack recent events unless connected to tools or retrieval. Models can memorize rare strings from the data, and the objective of predicting text does not guarantee truthfulness. Careful evaluation is needed, because benchmarks may overlap with training data.
Continued and domain-adaptive pretraining
In continued pretraining, a model that has been pretrained on general data is trained further on text from a specific domain, such as law, medicine or finance, or in a specific language. This can improve domain vocabulary and knowledge more deeply than prompting alone, at lower cost than starting from scratch. Developers must guard against forgetting, in which a model loses earlier abilities, by mixing in general data and by monitoring benchmarks.
A worked example
A research group wants a model that understands chemistry papers. They begin with an open pretrained language model and continue pretraining it on several billion tokens of open-access chemistry articles, with a small share of general text to prevent forgetting. They then fine-tune on a few thousand labeled examples for tasks such as extracting reaction conditions. The final model outperforms the general model on chemistry questions, and the group reports its data sources and limits.
Scaling laws and compute-optimal pretraining
A key finding of the early 2020s is that the loss of a pretrained model falls in a smooth, predictable way as model size, dataset size and computing power increase. These scaling laws let teams forecast the results of expensive runs from small experiments. A 2022 study by DeepMind, often called the Chinchilla paper, found that for a fixed compute budget, size and data should grow roughly together, and that many earlier large models were trained on too little data. Developers now often train somewhat smaller models on far more tokens, which reduces the cost of using the model later. Because high-quality text is finite, researchers are also studying how to repeat data, improve its quality and add synthetic examples without harming the model.
Pretraining beyond text
The same recipe works for other kinds of data. Image models learn by reconstructing masked patches or by matching images with captions, speech models learn from tens of thousands of hours of unlabeled audio, and robotics models pretrain on large collections of demonstrations. Scientific models pretrain on protein sequences, molecules or weather records. In each case, a general model is learned first and then adapted, which is why pretraining sits at the center of modern AI practice and of the broader idea of transfer learning.
Future directions
Current research looks at more efficient objectives, curricula that order data from easy to hard, better filtering that favors educational content, ways to update a model’s knowledge without full retraining and methods that reduce memorization of private data. Many researchers also expect more pretraining in multimodal form, so that one model learns jointly from text, images, audio and video.
FAQs
What is the difference between pretraining and fine-tuning? Pretraining learns general patterns from broad unlabeled data, while fine-tuning adapts the pretrained model to a particular task or style using smaller, targeted data.
Does pretraining need labeled data? Usually not, since self-supervised objectives use the data itself as the label.
Why is pretraining so expensive? It processes enormous datasets with very large models, which requires thousands of accelerators for weeks.
More in Large Language Models
Key facts
- Pretraining uses self-supervised objectives, such as predicting the next word or a masked word, so no human labels are needed.1
- BERT and GPT popularized pretraining followed by fine-tuning for language tasks in 2018.2
- Pretraining on large datasets gives models general knowledge that can be adapted to many tasks, a form of transfer learning.3
Related concepts
Quick checkWhat is the goal of pretraining?Show answer
To give a model broad general knowledge and skills by training it on huge amounts of unlabeled data, so it can later be adapted to specific tasks with little extra data.
Sources
- Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, https://arxiv.org/abs/1810.04805
- Radford et al., Improving Language Understanding by Generative Pre-Training, https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
- Bommasani et al., On the Opportunities and Risks of Foundation Models, https://arxiv.org/abs/2108.07258
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
