Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Validation Data

nounAI Data and Infrastructurealso validation set, dev set, development set

In one line

Validation data is a portion of a data set held apart from training and used during development to tune a machine learning model's settings, compare candidate models and decide when to stop training.

Validation data, also called the validation set or development set, is a portion of a data set that is held apart from training and used during model development to make decisions: which settings to use, which model to choose and when to stop training. It sits between training data, which the model learns from, and testing data, which gives a final unbiased score. All three are normally carved out of one dataset.

Why a validation set is needed

Every machine learning project involves choices that the training algorithm does not make by itself. These include the learning rate, the number of layers in a neural network, the strength of regularization, which features to use and which of several model types to deploy. Such settings are called hyperparameters. To pick good values, developers try several candidates and compare how well each performs on data the model did not train on.

If they used the training data to compare, the most complex model would always win, because it can memorize the examples. If they used the test data, they would gradually tailor the model to it, and the test score would no longer be an honest estimate. The validation set solves this by providing a separate yardstick for development decisions, leaving the test set clean for one final check [1].

Main uses

  • Hyperparameter tuning. Compare settings, using methods such as grid search, random search or Bayesian optimization.
  • Model selection. Choose among algorithms or architectures, such as a decision tree versus a neural network.
  • Early stopping. Monitor validation error during training and stop when it no longer improves, which guards against overfitting [2].
  • Feature selection. Decide which inputs help generalization.
  • Threshold setting. Choose the cutoff at which a classifier’s probability becomes a yes or no decision.
  • Diagnosing problems. Comparing training and validation errors reveals whether a model is underfitting, with high error on both, or overfitting, with low training error but high validation error.

Validation loss and learning curves

During training, practitioners plot the loss on the training and validation sets after each pass through the data. Training loss usually keeps falling. Validation loss falls at first and then flattens or rises once the model begins to memorize noise. The point where validation loss is lowest is a natural place to stop, and many frameworks save the model at that point and restore it afterward. The gap between the two curves is a simple, useful signal of overfitting.

Cross-validation

When data is limited, setting aside a fixed validation set wastes examples and can give noisy results. Cross-validation addresses this. In k-fold cross-validation, the training data is divided into k equal parts, called folds. The model is trained k times, each time using a different fold as the validation set and the remaining folds for training, and the k scores are averaged [1]. Typical values of k are 5 and 10.

Variants include:

  • Stratified k-fold. Keeps class proportions equal in each fold, helpful for imbalanced data.
  • Leave-one-out. Uses a single example as validation each time, accurate but expensive.
  • Time-series split. Always validates on data that comes after the training period.
  • Nested cross-validation. An inner loop tunes hyperparameters and an outer loop estimates performance, giving an unbiased result when tuning is heavy.

See also cross-validation and hyperparameter tuning.

Typical split sizes

A common arrangement is 60 to 80 percent of data for training, 10 to 20 percent for validation and 10 to 20 percent for testing. For very large data sets, smaller percentages suffice, because even one percent can contain many thousands of examples. For small data sets, cross-validation is preferred over a single split.

Pitfalls

  • Overfitting to the validation set. If one tries hundreds of configurations, the best validation score becomes optimistic, since some choices win by chance. This is why the final test set is needed.
  • Data leakage. Duplicates or related examples across training and validation inflate scores; group-aware splitting helps.
  • Preprocessing mistakes. Fitting scalers or encoders on all data before splitting leaks information.
  • Unrepresentative validation data. If it differs from real use, tuning will optimize for the wrong target.
  • Time leakage. Random splits on time-ordered data let the model peek at the future.

Validation in other fields

The word has other meanings. In software and engineering, validation means checking that a system meets the needs of its users. In statistics, model validation covers broader checks of fit and assumptions. In machine learning, the term usually refers to the held-out data described here.

Validation in deep learning and large models

In deep learning, validation data plays a constant role because training runs are long and expensive, and the choices are many. Practitioners watch validation metrics at regular intervals, called evaluation steps, and use them to adjust the learning rate schedule, to select the best saved checkpoint and to compare architectures. Large language models are often monitored on several validation sets at once, for example held-out text from different domains and languages, so that a gain in one area that causes a loss in another is noticed early.

When models are fine-tuned on small, specialized data, the validation set is especially important, because the risk of overfitting is high. A common approach is to keep a few hundred carefully chosen examples for validation and track whether the model still handles general tasks after tuning.

Good practice checklist

  • Split the data before any exploration or preprocessing, and fit all transformations on training data only.
  • Make the validation set resemble the data the model will meet in production, including its mix of easy and hard cases.
  • Use group-aware or time-aware splits whenever examples are related.
  • Limit how many configurations are tried against the same validation set, or refresh it from time to time.
  • Record the random seeds, split definitions and data versions so that results can be reproduced.
  • Once the design is final, confirm performance on the untouched test set, and consider retraining on the combined training and validation data for the production model.

FAQs

What is validation data? It is a portion of a data set used during development to tune settings and choose among models, separate from training and test data.

What is the difference between validation and test data? Validation data guides development decisions and may be used many times. Test data is used once at the end for an unbiased final score.

What is cross-validation? It is a method that rotates which part of the data is held out for validation, then averages the results, to make better use of limited data.

How much data should be used for validation? Often 10 to 20 percent, though large data sets can use less and small ones should use cross-validation.

What is early stopping? It is stopping training when validation error stops improving, to prevent overfitting.

Key facts

  • A validation set is used to estimate prediction error for model selection, while a separate test set is used to assess the chosen model1.
  • In k-fold cross-validation, data is divided into k parts, and each part serves once as the validation set while the rest is used for training1.
  • Early stopping halts training when error on validation data stops improving, which helps prevent overfitting2.

Related concepts

Quick checkWhat is validation data?Show answer

Validation data is a portion of a data set kept separate from training and used during development to tune hyperparameters, compare models and decide when to stop training, before the final model is judged on a separate test set.

Sources

  1. Trevor Hastie, Robert Tibshirani and Jerome Friedman, The Elements of Statistical Learning, Springer, https://hastie.su.domains/ElemStatLearn/
  2. Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, https://www.deeplearningbook.org/

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue