A random forest is an ensemble learning method that builds many decision trees on random samples of the data and combines their predictions, by majority vote for classification or by averaging for regression. A single decision tree is easy to understand but unstable and prone to overfitting. A forest of trees, each slightly different, cancels out much of that noise and gives accurate, robust predictions with very little tuning. Introduced by the statistician Leo Breiman in 2001, random forests became one of the most widely used algorithms in machine learning, especially for tabular data.
How a random forest works
The algorithm uses two sources of randomness to make the trees different from one another.
- Bootstrap sampling: for each tree, a random sample of the training data is drawn with replacement, so some examples appear several times and others, about a third, are left out. This is bagging, or bootstrap aggregating.
- Random feature selection: at each split in each tree, only a random subset of the features is considered, often the square root of the number of features for classification. This prevents a few strong features from dominating every tree.
Each tree is grown deep, usually without pruning. To predict, the input passes through every tree, and the forest combines their outputs. Because the errors of diverse trees are only partly correlated, the average has lower variance than any single tree, while the bias stays low.
History of random forests
The idea grew from several threads. Breiman introduced bagging in 1996. Tin Kam Ho proposed random decision forests in 1995 and the random subspace method in 1998, in which trees see random subsets of features. Amit and Geman explored randomized trees for shape recognition. Breiman combined these in 2001 and added tools such as out-of-bag error and variable importance. Adele Cutler worked with him, and Random Forests is a registered trademark of the two. The method was later adopted in nearly every machine learning library.
Out-of-bag error
Because each tree is trained on only about two thirds of the data, the remaining third can serve as a built-in test set for that tree. For each training example, the forest collects predictions only from trees that did not see it, and compares the combined prediction with the true label. The resulting error, called the out-of-bag error, is a good estimate of how the forest will perform on new data, often removing the need for a separate validation set or cross-validation.
Feature importance
Random forests report which features matter. Impurity-based importance totals how much each feature reduces impurity across all the splits where it is used, and it is quick but can favor features with many distinct values. Permutation importance shuffles one feature at a time and measures how much accuracy drops, which is more reliable. Newer tools such as SHAP values explain individual predictions. These measures help with feature selection and with understanding a model, but they show association, not causation.
Key hyperparameters
- Number of trees: more trees give more stable results, with diminishing returns, and rarely cause overfitting. A few hundred is common.
- Number of features per split: controls how different the trees are. Smaller values give more diversity.
- Maximum depth and minimum leaf size: limit tree complexity, speed up training and reduce memory.
- Class weights: help with imbalanced data.
Random forests work well with default settings, which is one reason for their popularity.
Advantages of random forests
- High accuracy on many problems and strong resistance to overfitting compared with single trees.
- Handle numerical and categorical features, missing values in some implementations and nonlinear relationships and interactions.
- Need no feature scaling and little tuning.
- Provide out-of-bag estimates and feature importance.
- Train in parallel, since trees are independent.
Disadvantages of random forests
The forest is less interpretable than one tree, since hundreds of trees cannot be read as a short rule set. It can be large in memory and slower to predict than a single model. It does not extrapolate beyond the range of training values in regression. On very high-dimensional sparse data such as text, linear models may do as well or better, and gradient boosting often achieves higher accuracy on tabular tasks, though with more tuning. Biased or unbalanced data produce biased forests.
Applications of random forests
Random forests are used for credit scoring, fraud detection, medical diagnosis and genetics, customer churn and propensity models, remote sensing and land cover classification, ecology and species distribution modeling, stock and demand forecasting and quality control. In computer vision, forests powered the body tracking in the Microsoft Kinect camera. They are often the first strong baseline that data scientists try on a new tabular problem.
Random forest vs. gradient boosting
Both combine trees, but in different ways. A random forest builds deep, independent trees in parallel and averages them, mainly to reduce variance. Gradient boosting builds shallow trees one after another, each fitting the errors of the previous ones, mainly to reduce bias. Boosting often wins in accuracy but needs more careful tuning and can overfit noisy data, while forests are more forgiving and easier to parallelize.
A worked example
A lender wants to predict loan defaults. The data scientist trains a forest of five hundred trees on ten years of loans with twenty features. The out-of-bag accuracy and area under the curve are close to those on a test set from the latest year. Permutation importance shows that the ratio of debt to income and the number of recent delinquencies matter most. The lender compares the forest with a logistic regression and a boosted model, chooses the best on validation data, and reviews outcomes by applicant group before use.
Practical tips
Start with a few hundred trees and the default feature setting, check the out-of-bag score and then tune depth and leaf size if needed. Use permutation importance, not impurity importance alone, when reporting which features matter. For imbalanced classes, set class weights or use balanced sampling, and judge the model with precision and recall. Save the trained forest together with the preprocessing steps so that new data are handled identically.
FAQs
Why is it called random forest? Because it is a forest of decision trees built with randomness in both the data samples and the features used.
Does a random forest overfit? Adding trees does not cause it to overfit, though very deep trees on noisy data can still fit noise.
Can random forests be used for regression? Yes. Each tree predicts a number, and the forest averages them.
More in Machine Learning
Key facts
- Random forests were introduced by Leo Breiman in 2001, building on bagging and the random subspace method.1
- Each tree is trained on a bootstrap sample and considers only a random subset of features at each split.1
- The out-of-bag error gives an internal estimate of accuracy without a separate validation set.2
Related concepts
Quick checkWhat two kinds of randomness make a random forest work?Show answer
Each tree is trained on a random bootstrap sample of the data, and at each split it considers only a random subset of the features, which makes the trees diverse.
Sources
- Breiman, Random Forests, https://link.springer.com/article/10.1023/A:1010933404324
- scikit-learn, Random forests, https://scikit-learn.org/stable/modules/ensemble.html#forest
- Wikipedia, Random forest, https://en.wikipedia.org/wiki/Random_forest
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
