Generalization is a machine learning model’s ability to perform well on new, unseen data, not just on the examples it was trained on. It is the whole point of learning from examples: a spam filter is useful only if it catches spam it has never seen, and a medical model matters only if it works on tomorrow’s patients. Everything else in model evaluation, from holding out testing data to guarding against overfitting, exists to measure and improve generalization.
Definition
Training data is a sample from some underlying distribution of possible inputs and outputs. A model’s generalization error, also called true or out-of-sample error, is its expected error on fresh examples from that distribution. It cannot be measured directly, so it is estimated on held-out data [1]. The generalization gap is the difference between training performance and held-out performance. A small gap with good absolute performance means the model has learned something real. A large gap signals overfitting, and poor performance on both signals underfitting.
Why generalization is hard
A model sees only a finite sample, so many functions fit the training data equally well. The learner must pick one that also works elsewhere, which requires assumptions, known as inductive bias. A linear model assumes straight-line relationships, a convolutional network assumes that nearby pixels matter and that patterns repeat across the image, and a transformer assumes that relationships between tokens can be learned through attention. No algorithm generalizes well on every problem, a point made formally by the no free lunch theorems. Success depends on matching the model’s biases to the structure of the task.
Theory in brief
Learning theory offers tools to bound the generalization gap [1].
- PAC learning asks how much data is needed so that, with high probability, a learned rule is approximately correct.
- VC dimension and Rademacher complexity measure a model class’s capacity; higher capacity needs more data to generalize.
- Bias-variance trade-off decomposes error into systematic error and sensitivity to the sample.
- Regularization and stability arguments show that methods which change little when one example changes tend to generalize.
These results work well for simple models but only loosely explain deep learning.
The puzzle of deep networks
Classical theory predicts that a network with more parameters than training examples should overfit badly. Yet large deep networks routinely reach zero training error and still generalize well. Zhang and colleagues showed that such networks can also memorize random labels perfectly, yet generalize on real labels, so capacity alone cannot explain the success [2]. Related observations include double descent, where test error falls, rises near the point where the model can just fit the data, then falls again as size grows, and the role of the optimization method, since gradient descent tends to find simple solutions among many that fit; see gradient descent. Research on this is ongoing.
Types of generalization
- In-distribution. New examples from the same source as training data. The standard meaning.
- Out-of-distribution. Data that differs from training, such as new hospitals, cameras, languages or time periods. Performance often drops; see data drift and model robustness.
- Compositional. Handling new combinations of known parts, such as unfamiliar sentences built from familiar words.
- Systematic or length generalization. Applying learned rules to longer or harder problems than seen in training.
- Zero-shot and few-shot. Doing tasks with no or few examples, a hallmark of large language models; see few-shot learning.
- Domain generalization. Training on several domains so that the model works on unseen ones.
How to measure it
Use separate validation data for tuning and untouched test data for the final estimate, apply cross-validation for small data sets, and make test sets resemble the deployment setting, including different groups and conditions. Out-of-distribution tests, stress tests and temporal splits check generalization beyond the training distribution. Evaluation of language models often uses fresh questions to avoid contamination; see AI benchmark.
How to improve it
- More and more diverse data, covering the cases the model will meet; see training data.
- Data augmentation to teach invariances.
- Regularization, dropout, weight decay and early stopping.
- Right-sized models and good inductive biases for the task.
- Transfer learning and pretraining, which bring broad knowledge.
- Ensembles, which average out individual quirks.
- Clean labels, since noise is memorized; see data labeling.
- Causal and invariant features, which stay valid when conditions change.
- Monitoring after deployment to catch decay.
Generalization failures
Models sometimes learn shortcuts, features that work in training but not in general. A classifier may identify cows by green pasture or pneumonia by a hospital marker. Such shortcut learning produces high test scores on data with the same quirks and failure on real inputs. Careful data design, error analysis and testing on varied sources reduce the risk.
Generalization in everyday terms
People generalize constantly. A child who has seen a few dogs can recognize a breed never met before, and a driver who has learned on one car can drive another. What makes human learning impressive is how few examples are needed and how well it carries over to new settings. Machine learning models are far hungrier for data and more brittle outside familiar conditions. A vision model trained on sunny daytime photos may fail at night or in fog, and a chatbot tuned on polite customer messages may stumble on slang. Closing this gap between narrow competence and flexible understanding is a central research goal, and it motivates work on foundation models, which learn broad knowledge from enormous and varied data before being adapted to specific tasks.
Practical guidance
A few habits protect generalization in real projects. Split data by the unit that will differ in deployment: if the model will see new patients, keep all records of a patient together in one split, and if it will see future data, split by time. Build test sets from the sources and conditions where the model will actually run, including the hard ones. Report performance by subgroup, not only on average; see algorithmic bias. Track the generalization gap during training, and treat a widening gap as a warning. After launch, compare live results with test results, since a drop signals that the world has moved away from the training distribution, a problem described under model drift.
FAQs
What is generalization in machine learning? It is how well a model performs on new, unseen data.
What is the generalization gap? The difference between a model’s performance on training data and on unseen data.
How do you improve generalization? Use more diverse data, regularization, augmentation, suitable model size, transfer learning and ensembles.
What is out-of-distribution generalization? Performing well on data that differs from the training distribution, such as a new environment.
Why do large neural networks generalize despite so many parameters? It is not fully understood, but optimization methods, architecture and data structure all play a role.
More in AI Evaluation and Reliability
Key facts
- Generalization error is the expected error of a model on new data drawn from the same distribution as the training data, estimated in practice on held-out test data1.
- The generalization gap is the difference between performance on training data and on unseen data1.
- Very large neural networks can fit their training data perfectly and still generalize, a finding that challenged classical views of overfitting2.
Related concepts
Quick checkWhat is generalization in machine learning?Show answer
Generalization is a model's ability to perform well on new, unseen data drawn from the same kind of source as its training data. It is the central goal of machine learning, as a model is only useful if what it learned from examples applies beyond them.
Sources
- Shai Shalev-Shwartz and Shai Ben-David, Understanding Machine Learning: From Theory to Algorithms, Cambridge University Press, https://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/
- Chiyuan Zhang et al., Understanding Deep Learning Requires Rethinking Generalization, International Conference on Learning Representations, https://arxiv.org/abs/1611.03530
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
