An attention mechanism is a technique in neural networks that lets a model decide which parts of its input matter most for each part of its output. When translating a long sentence, a person does not give equal weight to every word at every moment. They focus on the words relevant to the phrase they are writing. Attention gives networks the same ability, by assigning a weight to each input element and combining them according to those weights. It began as a fix for translation models and became the core idea of the transformer, the architecture behind modern large language models.
The problem attention solved
Early sequence-to-sequence models for translation used one recurrent neural network to read the whole source sentence and squeeze it into a single fixed-size vector, and another to write the translation from that vector. This bottleneck hurt performance on long sentences, because one vector could not hold all the detail. In 2014, Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio proposed letting the decoder look back at all the encoder’s states and learn which ones to focus on at each step. Translation quality improved markedly, especially for long inputs, and the idea spread quickly.
How attention works
The modern description uses three kinds of vectors: queries, keys and values. Think of a library search. The query is what you are looking for, each book has a key that describes it and a value that holds its contents. Attention compares the query with every key to get a relevance score, turns the scores into weights with a softmax so that they sum to one and returns the weighted average of the values. Books that match the query strongly contribute more to the result.
In the common scaled dot-product form, the score between a query and a key is their dot product divided by the square root of the key dimension, which keeps the numbers in a stable range. The output for each query is the sum of values weighted by the softmax of these scores. Because the weights are produced by the network and depend on the content, the model can focus differently on each input.
Types of attention
- Additive attention: the original Bahdanau form, which scores a query and key with a small feedforward network.
- Multiplicative or dot-product attention: scores with a dot product, which is faster and uses optimized matrix operations. Luong attention is an early example.
- Self-attention: queries, keys and values all come from the same sequence, so each position can look at every other position in that sequence.
- Cross-attention: queries come from one sequence, such as a decoder, and keys and values from another, such as an encoder, as in translation and in image generation guided by text.
- Multi-head attention: runs several attention operations in parallel with different learned projections, so that the model can attend to different kinds of relationships at once.
- Causal or masked attention: blocks positions from looking at later ones, which is needed when generating text one token at a time.
- Sparse and efficient attention: variants such as sliding windows, local attention and FlashAttention reduce the cost for long inputs.
Attention in the transformer
In 2017, Ashish Vaswani and colleagues at Google published Attention Is All You Need, which removed recurrence completely and built the transformer from stacked layers of multi-head self-attention and feedforward networks. Because every position attends to every other directly, the network handles long-range relationships easily and can process all positions in parallel, which makes training fast on modern hardware. The transformer, and attention at its heart, underlies BERT, the GPT family, Claude and nearly every current large language model, as well as vision transformers and many speech and protein models.
Visualizing attention
Attention weights can be drawn as a grid that shows which input elements each output element focused on. In translation, the pattern often lines up with the word alignment between languages. In a vision model, the weights can highlight the regions of an image that guided a decision. These pictures offer a glimpse into the model, but researchers warn that attention weights are not always a faithful explanation of why a model made a prediction, since other parts of the network also shape the result.
Applications of attention
Attention is used in machine translation, text summarization, question answering, chatbots, code generation, speech recognition, image captioning, image classification and generation, protein structure prediction as in AlphaFold, recommendation systems and time series forecasting. It also supports retrieval and memory features in language models, where a model attends to documents supplied in its context window.
Advantages of attention
- Direct connections between any two positions, which helps with long-range dependencies.
- Parallel computation across positions, which speeds up training.
- Flexible, content-based focus, learned from data.
- Some interpretability through attention maps.
- Works across modalities, including text, images, audio and graphs.
Limitations of attention
Standard attention compares every position with every other, so its cost grows with the square of the sequence length. This limits the context length and makes long documents and videos expensive, which motivates efficient attention methods. Attention has no built-in sense of order, so position information must be added through positional encodings. Large attention models need lots of data and computing power, and their behavior can be hard to interpret.
A worked example
Consider translating the sentence the cat sat on the mat into French. When the decoder is about to write the word for cat, its query matches most strongly with the key of the word cat in the source, so that word’s value dominates the weighted average and guides the output. When it writes the word for sat, the weights shift to sat. In a self-attention layer of a language model reading the sentence she put the book on the table because it was heavy, the word it can attend strongly to table or book depending on context, which helps the model resolve what the pronoun refers to.
Attention and memory in practice
In a large language model, attention is also how the model uses the text in its context window. Every new token is produced by attending over the prompt and earlier output, so instructions, examples and retrieved documents placed in the context can steer the answer. Engineers cache the keys and values of earlier tokens to avoid recomputing them, which speeds up generation but uses a great deal of memory for long contexts.
FAQs
Who invented the attention mechanism? Bahdanau, Cho and Bengio introduced it for translation in 2014, and Vaswani and colleagues made it the basis of the transformer in 2017.
What is multi-head attention? It runs several attention operations side by side, each learning to focus on different relationships, and combines their results.
Why is attention expensive for long inputs? Because every position is compared with every other, so cost grows with the square of the length.
More in Deep Learning and Neural Networks
Key facts
- Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio introduced attention for neural machine translation in 2014.1
- Attention computes a weighted average of values, with weights based on how well a query matches each key.2
- The 2017 paper Attention Is All You Need built the transformer entirely from attention and feedforward layers.2
Related concepts
In the news
- HistoryInternational Day for Monuments and Sites: Why Heritage Needs Guardians
- HealthWorld Haemophilia Day: Why Diagnosis Comes First
- HealthWorld Health Day
- SocietyInternational Day of Sport for Development and Peace: Play as a Path to Peace
- HealthWorld Autism Awareness Day: Acceptance, Inclusion and the Facts
Quick checkWhat are the three ingredients of an attention calculation?Show answer
Queries, keys and values. The query is compared with each key to produce weights, and those weights are used to average the values.
Sources
- Bahdanau, Cho and Bengio, Neural Machine Translation by Jointly Learning to Align and Translate, https://arxiv.org/abs/1409.0473
- Vaswani et al., Attention Is All You Need, https://arxiv.org/abs/1706.03762
- Lilian Weng, Attention? Attention!, https://lilianweng.github.io/posts/2018-06-24-attention/
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
