A decision tree is a supervised machine learning model that predicts an outcome by following a series of questions about the input, arranged like the branches of a tree. Each internal node asks a question about one feature, such as whether income is above fifty thousand. Each branch is an answer, and each leaf at the end gives a prediction, such as approve the loan. Decision trees can handle both classification, where the leaf gives a category, and regression, where the leaf gives a number. They are popular because their reasoning can be read and explained, and because they form the building blocks of powerful ensembles such as random forests.
Structure of a decision tree
A tree has a root node at the top, which holds the whole dataset and asks the first question. Internal nodes split the data further. Leaf nodes, or terminal nodes, stop the splitting and give the final answer. To predict, a new example starts at the root, follows the branch that matches its feature values at each node and reaches a leaf. The path from root to leaf reads as a rule, such as if age is over thirty and the number of late payments is zero, then predict low risk. A whole tree is therefore a set of human-readable rules.
How a decision tree learns
The learning algorithm builds the tree from the top down, a method called recursive partitioning. At each node it considers possible splits on each feature and picks the one that makes the resulting groups as pure as possible, meaning that each group is dominated by one class. It then repeats on each group. The process stops when a group is pure, when it is too small, when a maximum depth is reached or when further splits do not help. The main algorithms are ID3 by Ross Quinlan in 1986, its successors C4.5 and C5.0, and CART, short for classification and regression trees, by Leo Breiman and colleagues in 1984.
Measuring impurity
Several measures score how mixed a node is.
- Gini impurity: the chance that a randomly chosen item would be misclassified if labeled at random according to the node’s class proportions. It is zero for a pure node. CART uses it.
- Entropy and information gain: entropy measures disorder, and information gain is the reduction in entropy after a split. ID3 and C4.5 use them.
- Variance reduction: for regression, the split is chosen to minimize the variance of the target within the resulting groups.
In practice, Gini and entropy usually give similar trees.
A small example
Imagine predicting whether a person will play tennis based on the weather. The root might ask whether the outlook is sunny, overcast or rainy. If overcast, the answer is always yes, which is a pure leaf. If sunny, the tree might ask whether humidity is high, and if rainy, whether it is windy. Each question is chosen because it best separates the days when people played from the days when they did not. The final tree can be drawn on one page and read by anyone.
Overfitting and pruning
Left alone, a tree can keep splitting until every training example has its own leaf. Such a tree fits the training data perfectly but fails on new data, which is overfitting. Two strategies control this. Pre-pruning stops growth early by limiting the maximum depth, the minimum number of examples in a leaf or the minimum improvement from a split. Post-pruning grows a full tree and then removes branches that do not improve performance on validation data, as in cost-complexity pruning used by CART. Cross-validation is used to choose the settings.
Advantages of decision trees
- Interpretability: the rules are easy to explain to managers, regulators and patients.
- Little preparation: they do not need feature scaling and can handle numerical and categorical data.
- Nonlinear patterns and interactions: they capture relationships that linear models miss.
- Fast prediction: a prediction needs only a handful of comparisons.
- Feature importance: the tree shows which features matter most.
Disadvantages of decision trees
Single trees are unstable, because a small change in the data can produce a very different tree. They tend to overfit and often predict less accurately than ensembles. They create boundaries parallel to the feature axes, so they approximate diagonal boundaries with steps. They can be biased toward features with many possible values, and they do not extrapolate beyond the range of the training data in regression. Greedy splitting finds a good tree but not always the best one.
From trees to ensembles
The weaknesses of single trees led to ensemble methods, which combine many trees. Random forests train many trees on random samples and random subsets of features and average their votes. Gradient boosting builds trees in sequence, each correcting the errors of the previous ones, and libraries such as XGBoost, LightGBM and CatBoost dominate competitions on tabular data. These ensembles are far more accurate than single trees but less transparent, so teams sometimes use a small tree to explain the behavior of a larger model.
Applications of decision trees
Decision trees are used in credit approval, medical diagnosis and triage, customer churn prediction, fraud screening, quality control, marketing segmentation and troubleshooting guides. In business, decision trees are also drawn by hand to compare options under uncertainty, though that usage is related but different from the learned models described here. Their transparency makes them attractive where decisions must be justified.
A worked example
A clinic wants a tool that flags patients who may miss appointments. Using past records, the team builds features such as the days since booking, the number of earlier missed visits and the age group. A tree limited to a depth of four is trained. It finds that patients with two or more earlier missed visits and a long booking lead time are most likely to miss. Staff use the rule to send reminders and offer flexible slots to this group. The team checks that the rule does not unfairly burden any group and reviews it each quarter against new data.
FAQs
What is the difference between a decision tree and a random forest? A decision tree is one model, and a random forest averages many trees to improve accuracy and stability.
Do decision trees need scaling? No, since splits depend on the order of values and not their scale.
Are decision trees machine learning or statistics? Both. They are used in statistics and data mining as well as machine learning.
More in Machine Learning
Key facts
- A decision tree splits data by feature tests at internal nodes and gives predictions at leaf nodes.1
- Splits are chosen to reduce impurity, measured by Gini impurity or entropy for classification and variance for regression.1
- Trees can overfit, so they are limited by depth or pruned, and they are often combined into ensembles such as random forests.2
Related concepts
In the news
Quick checkHow does a decision tree choose where to split?Show answer
It tests possible splits and picks the one that most reduces impurity in the resulting groups, using measures such as Gini impurity or entropy.
Sources
- scikit-learn, Decision Trees, https://scikit-learn.org/stable/modules/tree.html
- Breiman et al., Classification and Regression Trees, https://www.routledge.com/Classification-and-Regression-Trees/Breiman-Friedman-Stone-Olshen/p/book/9780412048418
- Wikipedia, Decision tree learning, https://en.wikipedia.org/wiki/Decision_tree_learning
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
