A diffusion model is a type of generative model that creates new data by learning to undo a gradual noising process. During training, real examples, such as photographs, are corrupted with small amounts of random noise over many steps until nothing recognizable remains. The model learns to reverse each step. To generate something new, it starts from pure noise and removes noise step by step, guided by a prompt, until a clean image, sound or video emerges. Diffusion models power most leading systems for image generation, video generation and audio generation, and they are a central technique in generative AI.
History of diffusion models
The core idea came from physics. In 2015, Jascha Sohl-Dickstein and colleagues proposed a generative model inspired by nonequilibrium thermodynamics, in which structure in data is destroyed by diffusion and then recovered by a learned reverse process. The approach attracted little attention at first, since generative adversarial networks produced better images. In 2019, Yang Song and Stefano Ermon developed score-based models that learn the gradient of the data distribution. In 2020, Jonathan Ho, Ajay Jain and Pieter Abbeel published Denoising Diffusion Probabilistic Models, usually called DDPM, showing image quality competitive with GANs. In 2021, Dhariwal and Nichol reported that diffusion models beat GANs on image quality. The next years brought DALL-E 2, Imagen, Midjourney and Stable Diffusion, which made the method widely known.
How diffusion models work
The method has two processes.
- Forward process: a fixed procedure adds a small amount of Gaussian noise to a training example at each of many steps, often a thousand, until it becomes almost pure noise. This process needs no learning.
- Reverse process: a neural network learns to predict the noise present in a partly noised example at any step. Training consists of picking a random example, adding a random amount of noise, and asking the network to guess that noise, with the error measured by simple mean squared difference.
To generate, the system samples random noise and applies the network repeatedly, subtracting a little predicted noise each time. Because the steps are small, the network only has to make simple local corrections, which makes training stable. The network is usually a U-Net, a convolutional architecture, or a transformer in newer systems that use self-attention.
Conditioning and guidance
To make generation follow a text prompt, an embedding of the prompt is fed to the network, often through cross-attention. A technique called classifier-free guidance strengthens the influence of the prompt. The model is trained both with and without the prompt, and at generation time the two predictions are combined, with a guidance scale controlling how closely the result follows the text. Higher guidance gives results that match the prompt more, at the cost of variety.
Faster sampling and latent diffusion
The main drawback of early diffusion models was speed, since hundreds of network passes were needed. Researchers developed faster samplers such as DDIM and DPM-Solver, and distillation methods that compress many steps into a few. Latent diffusion runs the process in a compressed space made by an autoencoder, which sharply reduces computing needs and enabled models that run on consumer hardware.
Applications
Diffusion models create images from text, edit photographs, fill missing regions in inpainting, extend pictures in outpainting, increase image resolution and generate video and audio. They are also used in science, for example to design protein structures and molecules, and in robotics to plan movements. In medicine they can produce synthetic scans for training, and in synthetic data pipelines they create varied examples.
Comparison with other generative models
GANs generate in a single pass and are fast, but can be unstable to train and may miss parts of the data distribution. Variational autoencoders train stably but often give blurry outputs. Autoregressive models produce data piece by piece and work well for language. Diffusion models give high quality and diversity with stable training, but need many steps to generate, though faster samplers have narrowed that gap.
Limitations and concerns
Generation is slower and costlier than for single-pass models. Models can reproduce details of training images, which raises copyright and privacy concerns. They can show bias from their training data and can be misused for deepfakes. Controlling fine details and counting objects is still difficult. Training large models requires huge datasets and computing power, and generation uses energy.
A worked example
Suppose a model is asked for a red bicycle against a blue wall. It starts with a grid of random noise. In early steps, the network removes noise so that broad color regions emerge, with a red area against a blue background. In middle steps, shapes become clearer, a frame and two wheels appear, and in late steps fine detail, such as spokes and texture, is added. A different random seed gives a different bicycle in a different pose.
Training data and compute
Large diffusion models are trained on hundreds of millions to billions of image and caption pairs, often gathered from the web and filtered for quality and safety. Training can take weeks on clusters of graphics processors, and the data choices strongly shape what the model can draw and which biases it shows. Open models released with their weights, such as Stable Diffusion, let researchers and hobbyists fine-tune them on small datasets to learn a new style, character or product, using lightweight methods such as LoRA adapters. These adaptations are cheap, which has fueled both creative use and debates over consent and copyright.
Safety and provenance
Developers filter prompts and outputs, remove harmful material from training data and add watermarks or provenance metadata so that generated media can be identified. Researchers also study how to make models forget specific concepts or avoid copying training images. These safeguards are imperfect, so responsible use still depends on platform policies, clear labeling of synthetic content and human review of sensitive outputs.
Future directions
Current research aims at fewer sampling steps, better control over layout and identity, longer and more consistent video, joint generation of several media types and combining diffusion with language-model reasoning. Many systems already blend diffusion and transformers, and the line between the families of generative models continues to blur as methods are combined.
FAQs
Why is it called a diffusion model? The name comes from the physical process of diffusion, in which particles spread out randomly, like ink in water. The model learns to reverse a similar spreading of noise.
Are diffusion models better than GANs? For many image tasks they give higher quality and diversity and are easier to train, but they are slower at generation.
What is a U-Net in a diffusion model? It is the neural network, shaped like the letter U, that predicts the noise in each step.
More in Generative AI
Key facts
- Diffusion models learn to reverse a process that gradually adds Gaussian noise to training data until only noise remains.1
- The 2015 paper by Sohl-Dickstein and colleagues introduced the idea, and the 2020 DDPM paper made it practical for high-quality images.2
- Classifier-free guidance lets a text prompt steer generation without a separate classifier.3
Related concepts
In the news
Quick checkWhat does a diffusion model learn to do?Show answer
It learns to predict and remove the noise added to data, so that starting from pure random noise it can step by step produce a new, realistic sample.
Sources
- Ho, Jain and Abbeel, Denoising Diffusion Probabilistic Models, https://arxiv.org/abs/2006.11239
- Sohl-Dickstein et al., Deep Unsupervised Learning using Nonequilibrium Thermodynamics, https://arxiv.org/abs/1503.03585
- Ho and Salimans, Classifier-Free Diffusion Guidance, https://arxiv.org/abs/2207.12598
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
