An activation function is a mathematical function applied to the output of a neuron in a neural network. The neuron first computes a weighted sum of its inputs plus a bias, and the activation function then transforms that number into the neuron’s final output. Its most important job is to add nonlinearity. Without it, a network of many layers would behave like a single linear model, unable to learn curved boundaries or the rich patterns in images, speech and language. Choosing the right activation function affects how fast a network trains and how well it performs, so it is a basic design decision in deep learning.
Why nonlinearity matters
A linear function applied to a linear function is still linear. If every neuron only computed a weighted sum, stacking a hundred layers would give nothing more than one weighted sum with different weights. Such a network could separate two classes only with a straight line or flat plane and could not solve even simple problems like the exclusive or function. A nonlinear activation breaks this limit. With it, each layer can bend and fold the space, and many layers together can represent extremely complex functions. This is the reason the artificial neuron includes the activation step.
Common activation functions
### Step function
The step function outputs zero below a threshold and one above it. It was used in early perceptrons and mimics a neuron that either fires or does not. Its derivative is zero almost everywhere, so it cannot be trained with gradient descent, and it is rarely used today.
### Sigmoid
The sigmoid function squashes any number into the range between zero and one, with a smooth S-shaped curve. It is useful for outputs that are probabilities, such as in binary classification. As a hidden-layer activation, it has drawbacks. For large positive or negative inputs it flattens, so its gradient is nearly zero, which slows learning, and its outputs are not centered on zero.
### Hyperbolic tangent
The tanh function is similar to the sigmoid but ranges from negative one to one and is centered on zero, which often helps training. It still saturates for large inputs and can suffer from vanishing gradients.
### Rectified linear unit
The rectified linear unit, or ReLU, outputs zero for negative inputs and the input itself for positive ones. It is very cheap to compute, does not saturate for positive values and gives sparse activations, since many neurons output zero. It became the standard hidden-layer choice after its success in deep networks around 2011 and 2012. Its main weakness is the dying ReLU problem, in which a neuron whose input stays negative always outputs zero and stops learning.
### Leaky ReLU, PReLU and ELU
These variants fix the dying neuron problem by allowing a small slope for negative inputs. Leaky ReLU uses a fixed small slope, parametric ReLU learns the slope and the exponential linear unit curves smoothly for negative values.
### GELU and Swish
The Gaussian error linear unit, or GELU, and the Swish function, also called SiLU, are smooth curves that behave like ReLU for large inputs but pass small negative values. GELU is the standard choice in transformer models such as BERT and the GPT family, and Swish and its variants are common in image and language models.
### Softmax
Softmax is used in the output layer of multiclass classification. It converts a vector of raw scores into probabilities that add up to one, with larger scores getting larger probabilities. It is usually paired with the cross-entropy loss.
The vanishing and exploding gradient problem
Training uses gradients that flow backward through all layers by the chain rule, multiplying many derivatives together. If each derivative is smaller than one, as with saturated sigmoid or tanh units, the product shrinks toward zero in early layers, so they barely learn, which is the vanishing gradient problem. If derivatives are larger than one, gradients can blow up. ReLU and its relatives help because their derivative is one for positive inputs. Careful weight initialization, batch normalization and residual connections also help keep gradients healthy.
How to choose an activation function
A few rules of thumb guide the choice.
- For hidden layers in most feedforward and convolutional networks, start with ReLU, and try leaky ReLU or GELU if there are dead neurons or if the architecture recommends them.
- For transformers and many language models, use GELU or a gated variant such as SwiGLU.
- For a binary output, use sigmoid. For multiple exclusive classes, use softmax. For regression, use no activation, or a linear one, at the output.
- For recurrent networks, sigmoid and tanh are used inside LSTM and GRU gates, where bounded values are useful.
In the end, results are tested empirically on validation data, since the best choice depends on the architecture and the dataset.
Properties that matter
Activation functions differ in several ways. Differentiability matters because gradient descent needs derivatives. The output range affects stability, and zero-centered outputs often train faster. Monotonic functions are simple to reason about, though newer ones such as Swish are not strictly monotonic. Computational cost matters at scale, and ReLU is cheaper than sigmoid or tanh because it avoids exponentials. Smoothness can help optimization in very large models.
A worked example
Take a neuron whose weighted sum plus bias equals negative two. With a sigmoid, the output is about zero point one two. With tanh, it is about negative zero point nine six. With ReLU, it is zero, and with leaky ReLU using a slope of zero point one, it is negative zero point two. If the sum is three, the sigmoid gives about zero point nine five, tanh about zero point nine nine and ReLU three. The example shows how the same input can produce very different signals depending on the function, which affects how gradients flow during training.
Applications and trends
Activation functions appear in every neural network, from small classifiers to large language models. Research continues to produce new designs, including gated activations that multiply two linear projections, learnable activation functions and functions found by automated search, such as Swish. Hardware also matters, because simple functions run faster on accelerators and can be approximated in low-precision arithmetic.
FAQs
What is the most common activation function? ReLU is the most widely used for hidden layers, with GELU common in transformers.
Why is softmax used at the output? It turns raw scores into probabilities across classes that sum to one.
Can a network work without activation functions? It would still run, but it could only learn linear relationships, no matter how many layers it had.
More in Machine Learning
Key facts
- An activation function transforms a neuron's weighted sum into its output and makes the network nonlinear.1
- Common choices include sigmoid, tanh, ReLU, leaky ReLU, GELU and softmax.1
- Sigmoid and tanh can suffer from vanishing gradients in deep networks, which is one reason ReLU became the default hidden-layer activation.2
Related concepts
Quick checkWhy do neural networks need nonlinear activation functions?Show answer
Without them, any stack of layers would collapse into a single linear transformation and could not learn complex patterns.
Sources
- Stanford CS231n, Neural Networks Part 1: Activation Functions, https://cs231n.github.io/neural-networks-1/
- Goodfellow, Bengio and Courville, Deep Learning, https://www.deeplearningbook.org/
- Hendrycks and Gimpel, Gaussian Error Linear Units, https://arxiv.org/abs/1606.08415
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
