Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Embedding (Machine Learning)

nounLarge Language Modelsalso vector embedding, word embedding, text embedding

In one line

An embedding is a list of numbers, called a vector, that represents the meaning of a word, sentence, image or other item so that similar items sit close together.

An embedding is a list of numbers, called a vector, that represents the meaning or features of an item such as a word, sentence, image, song or user. Embeddings are learned by machine learning models so that similar items have vectors that lie close together in a high-dimensional space. The embedding for the word king sits near queen and monarch, and far from banana. Because computers can compare numbers quickly, embeddings turn fuzzy questions about meaning into simple geometry. They are a basic building block of deep learning, natural language processing, search, recommendations and large language models.

History of embeddings

The idea that words are known by the company they keep comes from linguists such as J. R. Firth in the 1950s. In the 1990s, latent semantic analysis represented words and documents as vectors from co-occurrence counts. In 2003, Bengio and colleagues learned word vectors within a neural language model. The field took off in 2013 when Tomas Mikolov and colleagues at Google released word2vec, whose vectors captured analogies, such as king minus man plus woman being close to queen. GloVe in 2014 and fastText in 2016 offered alternatives. Because these vectors give each word one meaning regardless of context, researchers developed contextual embeddings with ELMo and then BERT in 2018, in which a word’s vector depends on its sentence. Sentence-BERT in 2019 and later models produced embeddings for whole sentences and documents, and CLIP in 2021 learned a shared space for images and text.

How embeddings work

An embedding model maps an input to a vector with typically hundreds to thousands of dimensions. Each dimension has no simple human meaning, but together they encode properties such as topic, sense, sentiment and relationships.

  • Learning from context: in word2vec, a network learns to predict a word from its neighbors or the neighbors from the word, and the internal weights become the word’s vector.
  • Neural encoders: a transformer reads the whole text and outputs vectors. Sentence embeddings are often made by pooling token vectors or by using a special token, then training so that texts with related meaning are close and unrelated texts are far apart, a method called contrastive learning.
  • Joint embeddings: models such as CLIP embed images and captions in one space, so a photo and a matching sentence land near each other. See multimodal AI.
  • Learned embedding layers: inside language models, each token ID is mapped to a learned vector by an embedding table, which is trained together with the rest of the network.

Similarity between embeddings is usually measured with cosine similarity, which compares the angle between vectors, or with dot product or Euclidean distance.

Types of embeddings

  • Word embeddings: word2vec, GloVe and fastText.
  • Contextual embeddings: BERT and other transformer models, where meaning depends on the sentence.
  • Sentence and document embeddings: vectors for passages used in search.
  • Image, audio and video embeddings: vectors from vision and audio models.
  • Graph and knowledge embeddings: vectors for nodes in a network, such as users or products.
  • Multimodal embeddings: shared spaces across text, images and audio.

Applications

Semantic search finds documents by meaning rather than keywords. Retrieval-augmented generation embeds a document collection, retrieves passages close to a question and gives them to a language model to ground its answer. Recommendation systems embed users and items, so people are matched with similar items. Clustering and topic discovery group similar texts. Classification can be done by training a simple model on top of embeddings, and anomaly detection flags items far from the rest. Embeddings also support deduplication, translation, image search and face recognition. Storing and searching millions of vectors efficiently is the job of a vector database.

Strengths and limitations

Embeddings capture meaning flexibly, work across wording differences and allow fast comparison. However, they can encode biases from training data, such as gender stereotypes in word vectors. They are hard to interpret, can fail on rare terms, numbers and negation, and depend on the model and language, so embeddings from different models are not compatible and should not be mixed. Choice of dimension is a trade-off between detail and storage. They may also leak information about the text they represent, so sensitive embeddings need protection.

Evaluating embeddings

Benchmarks such as MTEB test embeddings on retrieval, clustering, classification and similarity tasks across languages. Practitioners also evaluate on their own data by checking whether relevant documents are retrieved for real queries. Quality can vary a lot by domain, so testing on your own examples is important.

Best practices

Use one embedding model consistently for a collection and for its queries, split long documents into sensible chunks, store metadata along with vectors, normalize vectors when using cosine similarity, re-embed when switching models and measure retrieval quality with a labeled set of questions. Consider hybrid search that combines keyword and vector matching.

A worked example

A company wants employees to search its policies. It splits each policy into paragraphs and uses an embedding model to turn each into a vector of 768 numbers, then stores them in a vector database. When an employee asks how many vacation days new staff get, the question is embedded and the database returns the closest paragraphs, which mention annual leave for new hires even though the word vacation does not appear. A language model then writes an answer using those paragraphs and cites them.

Visualizing and reducing embeddings

Because embeddings have hundreds of dimensions, people cannot look at them directly. Methods such as principal component analysis, t-SNE and UMAP project them down to two or three dimensions so that clusters become visible, for example groups of support tickets about the same problem. These plots are useful for exploration but can mislead, since distances in the projection are only approximate. For storage and speed, engineers shorten vectors with techniques such as Matryoshka representation learning, which trains the first dimensions to carry most of the meaning, and compress them by quantization, see quantization, trading a small loss in accuracy for large savings in memory.

Privacy, security and governance

Embeddings of private text can still reveal information, and research has shown that original text can sometimes be partly reconstructed from its embedding. Organizations should therefore treat embeddings of confidential material as confidential, control access to vector stores, delete vectors when the source documents are deleted and respect data protection rules about personal data. Poisoned or manipulated documents can also be crafted to rank highly in retrieval, so systems that feed retrieved passages to a model should filter sources and monitor for attacks.

Future directions

Research aims at embeddings that handle long documents and many languages, represent text, images, audio and video in one space, can be updated without recomputing everything and give more interpretable dimensions. Better evaluation on real tasks, rather than leaderboard averages, is also a priority, since the right embedding model depends heavily on the domain.

FAQs

What is the difference between an embedding and a vector? A vector is any list of numbers. An embedding is a vector learned to represent the meaning or features of an item.

How many dimensions does an embedding have? Commonly 384 to 3,072 for text, though it varies by model.

Can embeddings be used for images? Yes. Vision models produce image embeddings, and multimodal models align them with text.

Key facts

  • Word embeddings such as word2vec, published in 2013, represent words as vectors in which similar words are close together.1
  • Modern text and image embeddings are produced by neural networks and are used for search, recommendation, clustering and retrieval-augmented generation.2
  • Similarity between embeddings is usually measured with cosine similarity or dot product.3

Related concepts

Quick checkWhat is an embedding in machine learning?Show answer

It is a vector of numbers that represents an item, such as a word or image, so that items with similar meaning have vectors that are close together in space.

Sources

  1. Mikolov et al., Efficient Estimation of Word Representations in Vector Space, https://arxiv.org/abs/1301.3781
  2. Reimers and Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, https://arxiv.org/abs/1908.10084
  3. Jurafsky and Martin, Speech and Language Processing, chapter on vector semantics and embeddings, https://web.stanford.edu/~jurafsky/slp3/

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue