Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Latent Diffusion

nounGenerative AIalso latent diffusion model, LDM, Stable Diffusion architecture

In one line

Latent diffusion is a technique that runs the diffusion process in a compressed latent space instead of on full-size pixels, making image generation much cheaper and faster.

Latent diffusion is a way of running a diffusion model in a compressed space instead of on the pixels of an image. A pretrained autoencoder first squeezes an image into a small grid of numbers called a latent representation. The diffusion process, in which noise is added and then removed, takes place on that small grid, and a decoder turns the final result back into a full-size picture. Because the grid is many times smaller than the image, training and generation need far less memory and computing power. The approach was introduced in 2021 and is the basis of Stable Diffusion, which made high-quality image generation possible on a home computer.

History of latent diffusion

Early diffusion models for images, such as those described in the 2020 DDPM paper, worked directly on pixels. They produced excellent images but needed large computing budgets, since every step processed the full-resolution picture. In December 2021, researchers Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser and Bjorn Ommer from LMU Munich and Runway published High-Resolution Image Synthesis with Latent Diffusion Models. The work came out of the CompVis group and built on earlier ideas such as VQGAN. In August 2022, Stability AI with collaborators released Stable Diffusion with open weights, and it quickly spread to millions of users. Later versions, including SDXL and models using transformer backbones, improved image quality, text rendering and prompt following.

How latent diffusion works

A latent diffusion system has three main parts.

  1. Autoencoder: an encoder compresses an image into a latent tensor, for example a 512 by 512 pixel image becomes a 64 by 64 latent with a few channels, and a decoder reconstructs the image from it. The autoencoder, often a variational autoencoder, is trained first and then frozen. It is trained to keep perceptually important detail while discarding imperceptible detail.
  2. Diffusion network: a U-Net or transformer learns to remove noise from latents. Training adds noise to the latents of real images and asks the network to predict it.
  3. Conditioning encoder: a text encoder, such as CLIP or T5, converts the prompt into embeddings that the diffusion network reads through cross-attention layers. See attention mechanism.

To generate an image, the system draws a random latent, runs the diffusion network for a number of steps with the prompt embeddings as guidance and passes the final latent through the decoder.

Why working in latent space helps

Most pixel detail in a photograph is redundant or invisible to people, so a good autoencoder can keep the essential information in a much smaller form. A latent of about one forty-eighth the size lets the diffusion network spend its capacity on the meaning and layout of the image, while the decoder fills in fine texture. This reduces memory use, makes each step faster and lets researchers train bigger networks on the same hardware. It also makes it practical to generate high-resolution images on a consumer graphics card.

Conditioning and control

Latent diffusion supports many ways to steer generation. Text prompts are the most common, and classifier-free guidance controls how closely the result follows them. Other inputs include images, sketches, depth maps, edges and human poses, used through add-on networks such as ControlNet. Image-to-image generation starts from a noised version of an existing image, which preserves composition while changing style. Inpainting and outpainting change part of an image or extend it while keeping the rest. Small adapters such as LoRA and methods like DreamBooth teach a model a new subject or style from a handful of images.

Applications

Latent diffusion underlies many tools for illustration, concept art, advertising, photo editing, texture generation for games and design mock-ups. The same idea has been extended to video generation, where the latent includes a time dimension, and to audio generation, where a spectrogram is compressed before diffusion. Researchers also apply it to medical image synthesis, molecular design and super-resolution.

Strengths and limitations

The main strengths are efficiency, flexibility and open availability of models, which has produced a large community of tools and fine-tuned variants. Limitations include dependence on the autoencoder, since any detail it cannot represent, such as small text or fine faces, may be lost or distorted. Images can show artifacts at very high resolution. Models trained on web images raise copyright, privacy and bias concerns, and the ease of local use makes safety filters easier to remove in open models. See deepfake for related risks.

Comparison with pixel-space diffusion

Pixel-space models, such as the original DALL-E 2 decoder and Imagen, operate on full-resolution or cascaded low-resolution images and can capture fine detail directly, but need a lot of computing power. Latent diffusion trades some fidelity in fine detail for large gains in efficiency. Recent systems combine ideas, for example using stronger autoencoders with more channels or applying diffusion transformers to latents.

A worked example

A designer wants a poster background of misty mountains at dawn. She enters a prompt and a guidance scale of seven. The text encoder produces embeddings, and the system samples a random 64 by 64 latent, then runs thirty denoising steps in a few seconds. The decoder produces a 512 by 512 image. She likes the composition, so she upscales the latent to a larger size with image-to-image mode and a moderate noise strength, which adds detail without changing the layout, and finishes the poster in an editor.

Running latent diffusion locally

Because the models are small enough for a consumer graphics card with about eight gigabytes of memory, many people run latent diffusion on their own computers using open tools. Local use gives control over settings such as the random seed, the number of steps, the sampler and the guidance scale, and it keeps prompts and images private. Users can load community fine-tuned checkpoints, combine several LoRA adapters and chain tools for upscaling and face repair. The cost of this freedom is that the user is responsible for checking licenses, respecting the rights of others and not creating harmful content.

Related terms

The latent space of these models is also useful on its own. Because similar images sit near one another, researchers use it for interpolation between images, for searching collections and as a compact input for other models. It is a close relative of the representations used in foundation models and in multimodal AI systems that connect text and images.

FAQs

Is Stable Diffusion a latent diffusion model? Yes. Stable Diffusion is an open implementation of the latent diffusion approach.

What is the latent space? It is the compressed representation of an image learned by the autoencoder, much smaller than the pixel grid.

Does latent diffusion lower image quality? Slightly in fine detail, depending on the autoencoder, but the efficiency gains usually outweigh that cost.

Key facts

  • Latent diffusion models were introduced by Rombach and colleagues in 2021 and published at CVPR 2022.1
  • They apply diffusion in the latent space of a pretrained autoencoder, which sharply reduces computation compared with pixel-space diffusion.1
  • Stable Diffusion, released in August 2022, is the best-known open implementation.2

Related concepts

Quick checkWhy is latent diffusion cheaper than pixel-space diffusion?Show answer

It denoises a small compressed representation of the image instead of every pixel, so each step needs far less memory and computation, and a decoder restores the full-size picture at the end.

Sources

  1. Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, https://arxiv.org/abs/2112.10752
  2. Stability AI, Stable Diffusion Public Release, https://stability.ai/news/stable-diffusion-public-release

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue