Regression is a task in supervised learning in which a model predicts a continuous numerical value, such as a house price, tomorrow’s temperature or next month’s sales, from a set of input features. It differs from classification, where the output is a category. The word has an old origin. In the 1880s the statistician Francis Galton noticed that the children of very tall parents tended to be closer to average height, a pattern he called regression toward the mean, and the name stuck for the methods used to study relationships between variables. Regression is now one of the most widely used tools in machine learning, statistics, economics and science.
How regression works
Given a dataset in which each example has features and a numeric target, a regression algorithm learns a function that maps features to the target. The simplest form is a straight line. The algorithm adjusts the line’s parameters so that predictions are close to the actual values, measuring closeness with a loss function. The most common loss is squared error, and finding the parameters that minimize the sum of squared errors is called ordinary least squares, a method published by Adrien-Marie Legendre in 1805 and by Carl Friedrich Gauss soon after. In larger models, parameters are found by gradient descent. The result is tested on held-out data to check how well it predicts new cases.
Types of regression
- Simple linear regression: one input feature and a straight line, such as predicting salary from years of experience.
- Multiple linear regression: several input features combined linearly, such as predicting price from area, location and age.
- Polynomial regression: adds powers of the features so that the model can fit curves while remaining linear in its parameters.
- Ridge regression: adds a penalty on the size of the weights, called L2 regularization, which reduces overfitting and handles correlated features.
- Lasso regression: adds an L1 penalty that can set some weights exactly to zero, so it also selects features.
- Elastic net: combines the two penalties.
- Logistic regression: despite its name, used for classification, since it predicts probabilities of categories.
- Tree-based regression: decision trees and random forests predict by averaging the targets in regions of the data, and capture nonlinear patterns and interactions.
- Support vector regression, k-nearest neighbors and neural networks: flexible methods for complex relationships.
- Quantile and Poisson regression: specialized forms for predicting percentiles or counts.
Key assumptions of linear regression
Classic linear regression works best when certain conditions are met. The relationship between features and target should be roughly linear. The errors should be independent, have constant variance and be roughly normally distributed. The features should not be too strongly correlated with each other, a problem called multicollinearity. Violations do not always ruin predictions but can make coefficient estimates and confidence intervals unreliable. Analysts check residual plots, which show the errors, to see whether the assumptions hold, and they transform variables or choose other models when they do not.
Evaluating a regression model
Several measures summarize how far predictions are from actual values.
- Mean absolute error (MAE): the average size of errors, in the units of the target, and easy to interpret.
- Mean squared error (MSE) and root mean squared error (RMSE): penalize large errors more heavily.
- R-squared: the share of variation in the target that the model explains, from zero to one for a reasonable model.
- Mean absolute percentage error (MAPE): the average error as a percentage, useful when scale differs.
As in other tasks, results should be measured on data not used for training, and cross-validation gives a more stable estimate.
Overfitting, underfitting and regularization
A model that is too simple, such as a straight line through clearly curved data, underfits and has high bias. A model that is too flexible, such as a very high-degree polynomial, fits the noise in the training data and overfits, with high variance. The balance is the bias-variance trade-off. Regularization, more data, simpler models, feature selection and cross-validation help to find a good point. Ridge and lasso regression are standard ways of applying regularization.
Applications of regression
Regression predicts house and stock prices, energy and electricity demand, product sales, delivery times, crop yields, patient length of stay, insurance claim costs, ad click rates and weather variables. In science, it quantifies relationships, such as how dose relates to response. In economics, it estimates the effect of policies. In engineering, it calibrates sensors and models system behavior. Time series regression and forecasting models extend the idea to data ordered in time.
Advantages and limitations
Linear and related models are fast, transparent and need little data, and their coefficients show how each feature affects the prediction, which is valuable for explanation. Flexible models can capture complex patterns. Limitations include sensitivity to outliers in least squares, difficulty extrapolating beyond the range of the training data, the assumption of a fixed relationship that may change over time and the confusion of correlation with causation. A regression coefficient shows association, and it does not by itself prove that changing a feature causes a change in the target.
A worked example
A real estate company wants to estimate sale prices. It collects past sales with area, number of bedrooms, age, neighborhood and distance to transport. It cleans the data, encodes the neighborhood and scales numerical features, then fits a ridge regression and a gradient boosting model. On a held-out set, the boosted model has a root mean squared error that is fifteen percent lower. The team inspects the largest errors, finds a few luxury homes that the model underprices and adds features to capture them. They report prediction intervals along with estimates, so users see how uncertain each price is.
Regression in deep learning and forecasting
Neural networks perform regression by ending in a single output node with no squashing function, trained to minimize squared or absolute error. They are used to predict continuous quantities from images, such as the age of a person in a photograph, and from sensor streams. Time series forecasting adds the challenge that observations depend on earlier ones, so models use lagged values, seasonal terms and trends, and validation must respect time order by testing only on later data than the training period.
Common mistakes to avoid
Frequent errors include extrapolating far beyond the range of the training data, ignoring outliers that distort least squares, using too many correlated features and treating a high R-squared on training data as proof of a good model. Always check residuals, validate on unseen data and report prediction intervals so users understand uncertainty.
FAQs
What is the difference between regression and classification? Regression predicts a number, and classification predicts a category.
Is linear regression machine learning? Yes. It is a basic supervised learning algorithm, although it is also a classical statistical method.
How do I choose between regression models? Start with a simple linear model as a baseline, then try regularized and tree-based models, and compare them on validation data with a suitable error measure.
More in Machine Learning
Key facts
- Regression is a supervised learning task whose output is a continuous numerical value.1
- Linear regression fits a straight line or plane by minimizing the sum of squared errors, a method known as least squares.2
- Common error measures are mean absolute error, mean squared error, root mean squared error and R-squared.1
Related concepts
Quick checkWhat does linear regression minimize?Show answer
The sum of squared differences between the predicted and actual values, which is known as the least squares method.
Sources
- scikit-learn, Linear models, https://scikit-learn.org/stable/modules/linear_model.html
- Google, Machine Learning Crash Course: Linear regression, https://developers.google.com/machine-learning/crash-course/linear-regression
- Wikipedia, Regression analysis, https://en.wikipedia.org/wiki/Regression_analysis
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
