Ensemble learning is a machine learning approach in which several models, called base learners, are trained on the same problem and their predictions are combined into one final answer. The idea mirrors the wisdom of crowds. A group of reasonably good, diverse models will usually make fewer mistakes than any one of them alone, because their errors partly cancel. Ensembles power many of the most accurate systems in machine learning, from fraud detection to winning entries in data science competitions, and they are especially strong on structured, table-like data.
Why ensembles work
The errors of a model come from three sources. Bias is error from overly simple assumptions, variance is error from being too sensitive to the particular training data and noise is error that cannot be removed. Combining models can reduce variance, bias or both. If several models make different mistakes, averaging their outputs smooths out the individual errors. The key requirement is diversity. If all the models are identical, combining them adds nothing. Diversity is created by training on different samples of the data, using different features, using different algorithms or starting from different random settings.
Main ensemble strategies
### Bagging
Bagging, short for bootstrap aggregating and introduced by Leo Breiman in 1996, trains many models in parallel, each on a random sample of the training data drawn with replacement. For classification the models vote, and for regression their predictions are averaged. Bagging mainly reduces variance and works well with flexible models such as deep decision trees, which tend to overfit. The best-known example is the random forest, which adds random selection of features at each split.
### Boosting
Boosting trains models one after another, with each new model focusing on the examples that earlier ones got wrong. The final prediction is a weighted combination. AdaBoost, proposed by Yoav Freund and Robert Schapire, adjusts the weight of each training example. Gradient boosting, developed by Jerome Friedman, fits each new model to the remaining errors of the current ensemble. Libraries such as XGBoost, LightGBM and CatBoost are very popular implementations. Boosting mainly reduces bias, though it can overfit noisy data if run too long.
### Stacking
Stacking, or stacked generalization, trains different kinds of models, for example a logistic regression, a decision tree and a neural network, and then trains a final meta-model that learns how best to combine their outputs. To avoid leaking information, the meta-model is trained on predictions made for data the base models did not see during their training, usually through cross-validation. Stacking can capture the strengths of very different algorithms.
### Simple combination rules
The simplest ensembles use majority voting, in which each model casts one vote, or averaging of probabilities, sometimes with weights based on each model’s validation accuracy. Such blending is easy to build and often gives a worthwhile gain.
Applications of ensemble learning
Ensembles are used in credit scoring, insurance pricing, fraud detection, medical diagnosis, customer churn prediction and recommendation. Random forests and gradient boosting are standard choices for tabular business data. In weather and climate forecasting, ensembles of simulation runs with slightly different starting conditions give a range of possible outcomes and an estimate of uncertainty. In computer vision and speech, teams combine several neural networks for extra accuracy. The Netflix Prize, a contest that ended in 2009 to improve movie recommendations, was won by a blend of more than a hundred models, which helped popularize the technique.
Advantages of ensemble learning
- Higher accuracy: combined predictions are usually better than any single model.
- Robustness: results are less sensitive to noise and to the quirks of one training sample.
- Flexibility: the methods work with many kinds of base models and tasks.
- Uncertainty estimates: disagreement among members shows when a prediction is unreliable.
Disadvantages and challenges
Ensembles cost more. Training and running many models requires more computing time and memory, which can matter for real-time or mobile uses. They are also harder to interpret, since a forest of hundreds of trees is not as easy to explain as a single tree, although tools such as feature importance and Shapley values help. Boosted ensembles can overfit noisy labels, stacking is complex to build correctly, and gains shrink when the base models are too similar. For these reasons, teams weigh the extra accuracy against the cost and the need for explanation.
A simple example
Suppose a bank wants to predict loan default. It trains a logistic regression model, a random forest and a gradient boosting model on past loans. Each gets about the same accuracy, but they make mistakes on different borrowers. Averaging their predicted probabilities gives a combined score that is more accurate than any single model and gives fewer false alarms. The bank checks that the ensemble treats different groups of applicants fairly before using it, because a more accurate model can still be biased.
Ensemble learning vs. a single model
A single, well-tuned model is easier to explain, cheaper to run and often good enough. An ensemble is worth the effort when small gains in accuracy are valuable, the data are noisy, or reliable uncertainty estimates are needed. A sensible workflow is to build a simple baseline first, then try bagging or boosting and use stacking only if the extra complexity pays off.
Tips for building ensembles
A few practices make ensembles work better. Use a strong baseline first, so you know whether the added complexity pays off. Pick base models that differ, for example a tree-based model, a linear model and a neural network, since diversity is the source of the gain. Use cross-validation to produce honest predictions for stacking, and keep a final test set that no step has touched. Tune the number of models and the learning rate in boosting with early stopping, which halts training when validation accuracy stops improving. Finally, measure the cost. If an ensemble raises accuracy by a fraction of a percent but triples the serving time, a simpler model may serve users better.
Ensembles in deep learning
Ensembling also applies to neural networks. Teams train several networks from different random starting points and average their outputs, which usually improves accuracy and gives better estimates of uncertainty. Techniques such as dropout can be seen as a cheap ensemble inside a single network, and snapshot ensembles save several versions along one training run. Model soups average the weights of fine-tuned versions of one pretrained model. Some modern language model systems combine several models or several answers from one model by majority voting, a method called self-consistency, to improve reasoning accuracy.
FAQs
What is the difference between bagging and boosting? Bagging trains models independently in parallel and averages them to reduce variance. Boosting trains models in sequence so each corrects the last, which mainly reduces bias.
Is a random forest an ensemble? Yes. It is a bagging ensemble of decision trees with random feature selection.
Do ensembles always beat single models? Not always. They help most when the members are accurate and diverse, and they can be unnecessary for simple problems.
More in Machine Learning
Key facts
Related concepts
Quick checkWhat are the three main ensemble strategies?Show answer
Bagging, which trains models in parallel on resampled data; boosting, which trains models in sequence to fix earlier errors; and stacking, which learns how to combine different models.
Sources
- scikit-learn, Ensemble methods, https://scikit-learn.org/stable/modules/ensemble.html
- Wikipedia, Ensemble learning, https://en.wikipedia.org/wiki/Ensemble_learning
- Dietterich, Ensemble Methods in Machine Learning, https://link.springer.com/chapter/10.1007/3-540-45014-9_1
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
