Self-supervised learning is a machine learning approach in which a model creates its own training signal from unlabeled data, usually by hiding part of the input and learning to predict it. A language model, for example, reads a sentence with a word removed and tries to guess the missing word. Because the answer is already in the data, no human has to supply labels. This lets models learn from the huge amounts of text, images and audio available online, and it is the main technique behind modern large language models and many image and speech systems.
History of self-supervised learning
Ideas about learning from the structure of data are old, and early examples include autoencoders, which learn to reconstruct their own input, and word embedding models of the early 2010s, which learned word meanings from the words around them. The approach grew rapidly after 2018, when researchers showed that pretraining large networks on unlabeled text produced models that could be adapted to many tasks. Google’s BERT, introduced in 2018, learned by predicting masked words, and OpenAI’s GPT models learned by predicting the next word. In 2020, contrastive methods such as SimCLR made self-supervised learning competitive for images. Yann LeCun has been a prominent advocate, arguing that self-supervision is key to machines that learn efficiently, like animals and people.
How self-supervised learning works
The process has two stages. First, in pretraining, the model solves a pretext task built from the data. The task is chosen so that solving it requires understanding something useful about the data. Predicting a masked word needs knowledge of grammar and meaning, and predicting a hidden image region needs knowledge of objects and scenes. The model learns internal representations, numerical summaries of the data that capture its structure. Second, in adaptation, the pretrained model is fine-tuned on a smaller labeled dataset for a particular task, or used directly through prompting. The heavy learning happens in the first stage, which is why labeled data needs are modest in the second.
Main approaches
Several families of pretext tasks are used.
- Masked prediction: parts of the input are hidden and the model fills them in, as in BERT for text and masked autoencoders for images.
- Next-token prediction: the model predicts the next element of a sequence, which is how GPT-style language models are trained.
- Contrastive learning: the model learns to place different views of the same item, such as two crops of one photo, close together and views of different items far apart.
- Reconstruction and denoising: the model rebuilds the original data from a corrupted version.
- Predicting relationships: the model learns the order of frames in a video or whether two sentences follow each other.
Applications of self-supervised learning
Language is the most visible application. Chatbots, translators and search engines are built on models pretrained by predicting words. In computer vision, self-supervised pretraining helps with image classification, object detection and medical imaging, where labeled scans are scarce. In speech, models such as wav2vec learn from raw audio and then need only a little transcribed speech. Self-supervision is also used in biology to learn representations of proteins and genomes, in robotics to learn from video and in recommendation systems to learn from user behavior. It forms the basis of foundation models that can be adapted to many tasks.
Self-supervised vs. unsupervised and supervised learning
Self-supervised learning is often seen as a form of unsupervised learning, because no human labels are used, but it differs in an important way. Classic unsupervised learning looks for structure such as clusters, while self-supervised learning builds a prediction task with a definite right answer that is derived from the data, and it trains with the same loss functions and optimization as supervised learning. This framing lets researchers use powerful supervised training methods without paying for labels. In semi-supervised learning, by contrast, a portion of the data carries real human labels.
Advantages and limitations
The main advantage is scale. Since labels are not required, models can learn from billions of examples, producing general features that transfer to many tasks and reduce the labeled data needed for each. The main limitations are cost and risk. Pretraining large models takes enormous computing resources and energy. Models absorb the biases, errors and private information present in the raw data, and the quality of what they learn depends on the pretext task. Evaluating what a model has really learned is also difficult, since it was never trained on a single explicit goal.
Self-supervised learning in practice
A concrete case shows how the pieces fit. To train a language model, engineers collect a large body of text and split it into small units called tokens. At each position, the model is asked to predict the next token from the ones before it. When it is wrong, the training algorithm adjusts the model’s parameters to make the right token more likely. Repeating this across trillions of tokens teaches grammar, facts and patterns of reasoning, all without a single human-written label. In image work, a masked autoencoder hides most patches of a photo and asks the network to reconstruct them, which forces it to learn what objects and textures look like. After pretraining, a small labeled dataset can adapt the model to a specific job such as classifying medical scans.
Open questions and future directions
Researchers still debate which pretext tasks produce the most useful representations and how well the learned features transfer to new problems. Some argue that predicting the next word alone cannot yield a model of the physical world, and advocate learning from video and other sensory data. Others focus on efficiency, since training on ever larger datasets uses large amounts of energy. Questions of data provenance also matter, because web-scale training data can include copyrighted or personal material. Even so, self-supervised learning has become the default way to build general-purpose models, and progress in it is closely linked to progress in artificial intelligence as a whole.
FAQs
Is ChatGPT trained with self-supervised learning? Its base language model is pretrained by predicting the next token in text, a self-supervised task, and then refined with additional methods, including human feedback.
What is a pretext task? It is an artificial task, such as predicting a masked word, designed so that learning it produces useful representations of the data.
Why is self-supervised learning important? It allows models to learn from vast amounts of unlabeled data, which is much cheaper and more plentiful than labeled data.
More in Machine Learning
Key facts
- In self-supervised learning, labels are generated automatically from the data itself, often by hiding part of the input and asking the model to predict it.1
- BERT, introduced in 2018, was trained by predicting masked words in text, and GPT-style models are trained to predict the next token.2
- Contrastive methods such as SimCLR learn image representations by pulling together different views of the same image.3
Related concepts
Quick checkWhat is the training signal in self-supervised learning?Show answer
It comes from the data itself, such as a hidden word or image patch the model must predict, so no human labels are needed.
Sources
- Wikipedia, Self-supervised learning, https://en.wikipedia.org/wiki/Self-supervised_learning
- Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers, https://arxiv.org/abs/1810.04805
- Chen et al., A Simple Framework for Contrastive Learning of Visual Representations, https://arxiv.org/abs/2002.05709
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
