Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Data Preprocessing

nounAI Data and Infrastructurealso data preparation, data cleaning and transformation, data wrangling

In one line

Data preprocessing is the set of steps that clean, transform and organize raw data, such as handling missing values, scaling numbers and encoding categories, so that it is suitable for analysis and machine learning.

Data preprocessing is the group of steps that turn raw data into a clean, consistent form suitable for analysis and machine learning. Real-world data arrives with missing entries, inconsistent formats, duplicates, outliers and values on very different scales. Preprocessing repairs and reshapes it so that a learning algorithm can use it. In practice this stage often takes more time than building the model itself [1], and it strongly influences results. Preprocessing sits near the start of a data pipeline and prepares the dataset that becomes training data.

Why preprocessing matters

Models are sensitive to the quality and form of their inputs. A column measured in dollars alongside another measured in fractions can let the larger numbers dominate. Text containing stray symbols, or dates written five different ways, can confuse a system. Errors that go in produce errors that come out, often summarized as “garbage in, garbage out.” Good preprocessing improves accuracy, speeds training and makes models more stable.

Main steps

### Data cleaning

  • Remove duplicates. Repeated records bias training and can leak into test sets.
  • Fix errors and inconsistencies. Typos, mixed units, impossible values such as an age of 250, and differing spellings of the same category.
  • Handle missing values. Options include deleting rows or columns, filling with the mean, median or most common value, using model-based imputation, or adding an indicator that a value was missing.
  • Handle outliers. Decide whether extreme values are errors or real. Methods include capping, transforming or using robust models.

### Data transformation

  • Scaling. Putting numeric features on comparable ranges. Standardization subtracts the mean and divides by the standard deviation. Min-max scaling maps values to a fixed range such as 0 to 1. Distance-based and gradient-based methods need this [2].
  • Normalization of distributions. Log or power transforms reduce skew.
  • Encoding categories. One-hot encoding creates a column per category, ordinal encoding maps ordered categories to numbers, and embeddings learn dense vectors for high-cardinality categories.
  • Discretization. Grouping numbers into bins.
  • Date and time features. Extracting day of week, month or elapsed time.

### Data reduction and feature work

  • Feature selection. Dropping irrelevant or redundant variables.
  • Dimensionality reduction. Methods such as principal component analysis compress many features into fewer; see dimensionality reduction.
  • Feature creation. Building new variables, such as ratios or aggregates; see feature engineering.
  • Sampling and balancing. Under- or oversampling to handle skewed classes.

Preprocessing for different data types

  • Text. Lowercasing, removing markup, tokenization, handling punctuation and splitting words into subword units; see tokenization and natural language processing.
  • Images. Resizing, cropping, converting color formats, normalizing pixel values and augmentation; see computer vision and data augmentation.
  • Audio. Resampling, trimming silence, converting to spectrograms.
  • Time series. Aligning timestamps, filling gaps, smoothing and creating lagged features.
  • Geospatial and sensor data. Coordinate transformations and calibration.

Avoiding data leakage

A frequent mistake is to calculate statistics, such as the mean used for scaling, on the whole data set before splitting it. This lets information from the test set seep into training and inflates results. The correct order is to split first, fit each preprocessing step on the training portion only, and then apply the learned transformation to validation and test data [2]. Bundling steps into a single pipeline object, as libraries like scikit-learn allow, makes this easier and keeps the same steps in training and production. See testing data and validation data.

Tools

Common tools include Python libraries such as pandas for tables, NumPy for arrays and scikit-learn for transformations, SQL for databases, and workflow tools and platforms for large-scale processing. Distributed systems handle data too big for one machine; see distributed computing.

Best practices

  • Look at the data first with summaries and plots before changing it.
  • Keep raw data untouched and write transformations as code, so that they are repeatable.
  • Document every decision, such as how missing values were filled.
  • Apply identical steps in training and in production to avoid mismatch.
  • Monitor incoming data after deployment, because changes in input patterns can break assumptions; see data drift.

Preprocessing for modern deep learning

Deep networks can learn many features by themselves, but they still need clean inputs. Images are resized and normalized, text is tokenized, and large language models rely on heavy filtering and deduplication of their text collections before training.

A worked example

Imagine a bank building a model to predict which customers will miss a loan payment. The raw table has columns for age, income, employment type, postal code, number of late payments and the date the account opened. Preprocessing might proceed as follows. First, remove duplicate customer rows and correct impossible values, such as negative incomes. Second, fill missing income values with the median income of similar customers, and add a column noting that the value was filled in, because missingness can itself be informative. Third, convert the account opening date into the account’s age in months. Fourth, one-hot encode employment type and replace the high-cardinality postal code with broader regional groups. Fifth, split the table into training, validation and test portions. Sixth, fit a scaler on the training portion and apply it to all three. Each of these choices is recorded in code so that the same steps run when the model is later used on new applicants.

Common mistakes

  • Dropping too much. Deleting every row with a missing value can remove a large and unrepresentative part of the data.
  • Fitting on all data. Leakage of test information into training, as described above.
  • Inconsistent production steps. A model trained on scaled data but fed unscaled values in production produces nonsense. Packaging preprocessing with the model prevents this.
  • Hiding bias. Removing a sensitive column does not remove bias if other columns, such as postal code, act as proxies for it; see data bias.
  • Over-cleaning. Removing genuine rare events as if they were errors, such as a real fraud case that looks like an outlier.

Teams that automate these steps in reusable, tested pipelines, with checks that flag unexpected values, spend less time on rework and ship more reliable models.

FAQs

What is data preprocessing? It is the cleaning, transformation and organization of raw data so that it is ready for analysis and machine learning.

Why is data preprocessing important? Poor or inconsistent data produces poor models, and good preprocessing improves accuracy and stability.

What is the difference between normalization and standardization? Normalization usually rescales values to a fixed range such as 0 to 1, while standardization centers them at zero with unit variance.

How should missing values be handled? By removing them, filling them with a typical value or model estimate, or flagging them, depending on how much is missing and why.

What is data leakage in preprocessing? It is letting information from test data influence training, for example by scaling before splitting.

More in AI Data and Infrastructure

Key facts

  • Data preparation, including cleaning and transformation, commonly takes a large share of the time in a machine learning project1.
  • Many algorithms, such as k-nearest neighbors and gradient descent based models, are sensitive to the scale of input features, so scaling is a standard step2.
  • Preprocessing steps that learn parameters, such as the mean used for scaling, must be fitted on training data only to avoid data leakage2.

Related concepts

Quick checkWhat is data preprocessing?Show answer

Data preprocessing is the set of steps that clean, transform and organize raw data, such as filling in missing values, scaling numbers and encoding categories, so that it can be used effectively by analysis methods and machine learning models.

Sources

  1. Jiawei Han, Micheline Kamber and Jian Pei, Data Mining: Concepts and Techniques, Morgan Kaufmann, https://www.sciencedirect.com/book/9780123814791/data-mining-concepts-and-techniques
  2. scikit-learn developers, Preprocessing data, scikit-learn User Guide, https://scikit-learn.org/stable/modules/preprocessing.html

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue