Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Long Short-Term Memory

nounDeep Learning and Neural Networksalso LSTM, LSTM network, LSTM cell

In one line

Long short-term memory is a type of recurrent neural network that uses gates and a cell state to remember information over long sequences.

Long short-term memory, abbreviated LSTM, is a type of recurrent neural network that can learn and remember patterns across long sequences. It was designed to fix a flaw of plain recurrent networks, which tend to forget information from many steps back because the learning signal fades as it travels through time. An LSTM adds a separate memory line, called the cell state, and a set of gates that decide what to forget, what to store and what to output. For roughly two decades, LSTMs were the leading tool for speech recognition, translation and handwriting recognition, and they are still used in time series work.

History of LSTM

Sepp Hochreiter identified the vanishing gradient problem in his 1991 diploma thesis, under the supervision of Jurgen Schmidhuber. The two published the LSTM in 1997 in the journal Neural Computation. A key element, the constant error carousel, lets error signals flow back through many steps without shrinking. In 1999, Felix Gers, Schmidhuber and Fred Cummins added the forget gate, which allows the cell to reset itself, and this version is the one most people use. Later work by Alex Graves and others showed strong results in handwriting and speech, and from 2014 onward, LSTM-based systems were adopted by Google, Apple, Microsoft and Amazon for translation and voice assistants.

How an LSTM cell works

At each time step, an LSTM cell receives the current input and the previous hidden state, and it maintains a cell state that runs through the sequence like a conveyor belt with only small, controlled changes. Three gates, each a small neural layer with a sigmoid output between zero and one, regulate the flow.

  • Forget gate: looks at the input and previous hidden state and decides how much of each element of the old cell state to keep. A value near zero erases it, and a value near one keeps it.
  • Input gate: decides how much of the new candidate information to write into the cell state. A tanh layer proposes the candidate values.
  • Output gate: decides how much of the updated cell state to reveal as the new hidden state, which is passed on and used for predictions.

Because the cell state is updated mainly by addition and gentle multiplication, gradients can travel back through many steps, which solves much of the vanishing gradient problem.

An intuition

Think of reading a long novel. You do not remember every word, and you do not forget everything at each sentence. You keep track of the main characters and plot, drop details that stop mattering, add new facts when they appear and decide what to recall when asked a question. The forget gate drops, the input gate adds and the output gate recalls. The cell state is the running summary of the story.

Variants of the LSTM

  • Peephole connections: let the gates look at the cell state directly.
  • Bidirectional LSTM: runs one LSTM forward and another backward so each position sees past and future context. Used in tagging and speech recognition.
  • Stacked LSTM: several layers deep for more abstract features.
  • Convolutional LSTM: replaces matrix products with convolutions for spatial data such as weather maps and video.
  • Gated recurrent unit: a simpler cousin with fewer gates and parameters.

Applications of LSTMs

LSTMs have been used for speech recognition and synthesis, machine translation, text generation, language modeling, handwriting recognition and generation, music composition, video captioning, protein and genomic sequence analysis, sensor-based activity recognition, anomaly detection in machines and networks, and time series forecasting of demand, energy and prices. In 2016, Google Neural Machine Translation used a deep LSTM encoder and decoder with an attention mechanism. Many early virtual assistants relied on LSTMs for recognizing speech.

Advantages of LSTMs

  • Capture long-range dependencies better than plain recurrent networks.
  • Handle variable-length sequences and stream data step by step.
  • Robust training with gradient clipping and standard optimizers.
  • Well-tested, with mature support in major frameworks.

Limitations of LSTMs

LSTMs are slower to train than transformers, because each step depends on the previous one, so they cannot be parallelized across time. They have many more parameters than simple recurrent networks, and their memory, though better, is still limited for very long sequences, such as whole books. Transformers with attention connect any two positions directly and scale better with data and hardware, and they replaced LSTMs in most language tasks after 2017. LSTMs remain attractive for small datasets, low-latency streaming and devices with limited resources.

Tips for using LSTMs

Scale the input data, clip gradients, start with one or two layers and a moderate hidden size and use dropout between layers to prevent overfitting. Initialize the forget gate bias to a positive value so the cell remembers by default early in training. For time series, use windows of past values and test on later data than you trained on. Compare against simpler baselines, since a well-tuned linear model or gradient boosting model sometimes does as well.

A worked example

Suppose an energy company wants to forecast hourly electricity demand for the next day. It builds sequences of the last one hundred and sixty-eight hours of demand, temperature and calendar features, and feeds them to a two-layer LSTM, followed by a dense layer that outputs the next twenty-four values. The LSTM learns daily and weekly cycles and the effect of temperature. Trained on several years of data and tested on the most recent months, it beats a seasonal baseline. The company monitors errors during holidays and heat waves, where patterns change, and retrains regularly.

LSTM compared with plain RNNs and GRUs

A plain recurrent network has a single hidden state that is rewritten completely at every step, so old information is easily overwritten. An LSTM separates long-term storage in the cell state from the short-term hidden state and controls both with gates. A gated recurrent unit merges these into one state with two gates, so it has fewer parameters and trains a little faster, and in many tasks the two perform about equally. In practice, teams try both and choose the one that works better on validation data.

FAQs

What does LSTM stand for? Long short-term memory.

Why is it called long short-term memory? It provides a short-term memory, the hidden state, that can last for a long time thanks to the cell state and gates.

Are LSTMs obsolete? Not entirely. Transformers dominate language, but LSTMs are still used for time series, streaming and small-device tasks.

Key facts

  • LSTM was introduced by Sepp Hochreiter and Jurgen Schmidhuber in 1997 to address the vanishing gradient problem in recurrent networks.1
  • An LSTM cell has a cell state and three gates, the forget gate, the input gate and the output gate.2
  • LSTMs powered many speech recognition and machine translation systems before transformers became dominant.3

Related concepts

Quick checkWhat problem was the LSTM designed to solve?Show answer

The vanishing gradient problem, which stops plain recurrent networks from learning dependencies across many time steps.

Sources

  1. Hochreiter and Schmidhuber, Long Short-Term Memory, https://www.bioinf.jku.at/publications/older/2604.pdf
  2. Olah, Understanding LSTM Networks, https://colah.github.io/posts/2015-08-Understanding-LSTMs/
  3. Sutskever, Vinyals and Le, Sequence to Sequence Learning with Neural Networks, https://arxiv.org/abs/1409.3215

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue