Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Testing Data

nounAI Data and Infrastructurealso test set, test dataset, holdout set

In one line

Testing data is a held-out set of examples, never used during training or tuning, that is used to measure how well a finished machine learning model performs on new, unseen inputs.

Testing data, also called the test set or holdout set, is a collection of examples kept apart from a machine learning model’s development and used only to evaluate the finished model. Because the model has never seen these examples during training or tuning, its performance on them gives an honest estimate of how it will behave on new inputs in the real world [1]. Testing data is one of three standard portions of a dataset, along with training data and validation data.

Purpose

A model can score very well on the examples it learned from simply by memorizing them, a failure known as overfitting. High accuracy on training data therefore says little about real-world usefulness. The test set answers the question that matters: how well does the model generalize to data it has not seen? Reporting results on a proper test set is the basis of credible claims about a model, in research papers, product decisions and regulatory submissions.

The central rule is that the test set must stay untouched until the end. If developers look at test results repeatedly and adjust the model in response, information leaks from the test set into the design, and the final score becomes optimistic. This is why a separate validation set is used for tuning and model selection, with the test set reserved for one final check [1].

How data is split

Common practice is to divide the data randomly into three parts.

  • Training set. Typically 60 to 80 percent, used to fit the model.
  • Validation set. Often 10 to 20 percent, used to tune settings and compare candidate models.
  • Test set. Often 10 to 20 percent, used for the final evaluation.

The right proportions depend on how much data exists. With millions of examples, a test set of one percent may be plenty. With small data sets, a single split is unreliable, so practitioners use cross-validation, which rotates which portion is held out and averages the results, often with a final untouched test set.

### Splitting strategies

  • Random split. Suitable when examples are independent.
  • Stratified split. Keeps the proportions of each class the same in every portion, important for imbalanced data.
  • Time-based split. For forecasting, trains on the past and tests on the future, to avoid looking ahead.
  • Group-based split. Keeps all records from one person, device or document together, so near-duplicates do not appear on both sides.

Data leakage

Data leakage happens when information that would not be available in real use finds its way into training, producing inflated test scores [2]. It is among the most common and costly mistakes in machine learning. Examples include:

  • Duplicates across splits. The same or near-identical example in both training and test sets.
  • Preprocessing leakage. Computing scaling or feature statistics on the whole data set before splitting.
  • Target leakage. Using a feature that is only known after the outcome, such as a treatment code that reveals a diagnosis.
  • Temporal leakage. Using future information to predict the past.
  • Benchmark contamination. For large language models trained on web text, benchmark questions and answers may appear in the training corpus, making scores unreliable; see LLM evaluation.

Guarding against leakage means splitting first, fitting every transformation on training data only, deduplicating, and thinking carefully about what information exists at prediction time.

Metrics on the test set

The test set is scored with metrics suited to the task, such as accuracy, precision, recall, F1 score and area under the curve for classification, and mean squared error for regression. Choice of metric matters: on a data set where 99 percent of cases are negative, a model that always predicts negative has 99 percent accuracy but is useless. It is good practice to report confidence intervals, examine errors by subgroup to check fairness, and look at a confusion matrix in addition to a single number. See model evaluation.

Representativeness and distribution shift

A test set is only useful if it resembles the situations the model will face. A model tested on clean, curated photographs may fail on blurry phone images. Real-world data also changes over time, a problem called distribution shift or data drift, so a model that scored well at launch can degrade. Good practice includes building test sets that cover edge cases and different user groups, refreshing them, and monitoring performance after deployment.

Benchmarks and label quality

Shared public test sets, called benchmarks, let researchers compare models fairly, but they have limits. Models can be tuned for a benchmark until it stops measuring general ability, and test labels themselves contain errors. Northcutt and colleagues found label errors in the test sets of ten widely used data sets and showed that these errors could change which model appears best [3]. Careful curation and periodic benchmark renewal are therefore important.

Testing data beyond machine learning

The term is also used in software engineering, where test data are inputs designed to check that a program behaves correctly. In machine learning the sense is related but distinct: the test set measures statistical performance, not just correctness of code.

Testing data in practice

Teams that evaluate models well tend to follow a few habits. They define the test set before building the model, and write down how it was collected, so that later changes cannot quietly make the numbers look better. They store the test set in a read-only location and limit who can use it. They keep a record of how many times each test set has been used, because every look at the results is a small chance to over-adapt the model to it.

They also build slice-based tests, which score the model separately on important subsets, such as rare classes, different languages, lighting conditions or customer groups. An overall accuracy of 95 percent can hide a failure rate of 30 percent on a small but important group. Finally, they run stress tests and adversarial tests, using inputs that are deliberately unusual or hard, to see where the model breaks. These checks complement the standard test set and are essential for safety-critical systems such as medical or driving software.

When a model is retrained, the test set is usually kept fixed so that scores from different versions can be compared. Over time, however, a fixed test set can drift away from reality, so mature teams maintain a fresh “canary” set drawn from recent production data and use it alongside the stable one.

FAQs

What is testing data? It is a held-out set of examples used only at the end to measure how well a finished model performs on unseen inputs.

What is the difference between validation data and testing data? Validation data is used during development to tune and compare models. Testing data is kept untouched until a final evaluation.

How big should a test set be? Often 10 to 20 percent of the data, but it depends on data volume. It must be large enough to give a stable estimate.

What is data leakage? It is when information from outside the training data, such as test examples, influences the model, causing overly optimistic results.

Can I train on my test set after evaluating? Yes for a final production model, once evaluation is done, but then you no longer have an unbiased estimate for that model.

More in AI Data and Infrastructure

Key facts

  • A test set is held back from model development so that it gives an unbiased estimate of how the final model performs on unseen data1.
  • Data leakage, where information from outside the training data influences the model, can make results look better than they will be in real use2.
  • Studies of popular benchmark data sets found label errors in their test sets, which can change the ranking of models3.

Related concepts

Quick checkWhat is testing data?Show answer

Testing data is a set of examples kept separate from training and tuning and used only at the end to measure how well a finished machine learning model performs on new, unseen inputs.

Sources

  1. Trevor Hastie, Robert Tibshirani and Jerome Friedman, The Elements of Statistical Learning, Springer, https://hastie.su.domains/ElemStatLearn/
  2. Shachar Kaufman et al., Leakage in Data Mining: Formulation, Detection, and Avoidance, ACM Transactions on Knowledge Discovery from Data, https://doi.org/10.1145/2382577.2382579
  3. Curtis Northcutt, Anish Athalye and Jonas Mueller, Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, https://arxiv.org/abs/2103.14749

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue