Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

Data Pipeline

nounAI Data and Infrastructurealso ETL pipeline, ELT, data workflow

In one line

A data pipeline is an automated series of steps that collects data from sources, processes and moves it, and delivers it to a destination such as a database, analytics tool or machine learning model.

A data pipeline is an automated series of steps that collects data from one or more sources, processes it and delivers it to a place where it is used. The destination might be a data warehouse, a business dashboard or a machine learning model. Pipelines replace manual copying and spreadsheet work with repeatable, monitored workflows, and they are the plumbing behind nearly every data-driven product. In AI, they feed training data to models and supply fresh inputs to deployed systems; see model deployment.

Basic components

  • Sources. Databases, application logs, sensors, web services, files, third-party data feeds and user activity streams.
  • Ingestion. Moving data from sources into the pipeline, by scheduled pulls, change capture or event streams.
  • Processing and transformation. Cleaning, validating, joining, aggregating and reshaping; see data preprocessing.
  • Storage. Databases, data warehouses, data lakes and object stores.
  • Orchestration. Scheduling tasks, managing dependencies and retrying failures.
  • Serving and consumption. Delivering data to reports, applications, feature stores and models.
  • Monitoring. Tracking freshness, volume, errors and quality.

ETL and ELT

Two patterns dominate. In ETL (extract, transform, load), data is extracted from sources, transformed on a processing server and then loaded into the target, usually a data warehouse [1]. In ELT, raw data is loaded first into a scalable store and transformed inside it, using the storage system’s own computing power. ELT became popular with cloud data warehouses, which can handle large transformations cheaply, and it keeps raw data available for future reuse. Which is better depends on data volume, privacy needs and team skills.

Batch and streaming

  • Batch pipelines process data in chunks on a schedule, such as hourly or nightly. They are simple and efficient for reports and for retraining models.
  • Streaming pipelines process events continuously, within seconds or less, for uses such as fraud detection, recommendations and monitoring. They need systems that handle ordering, late data and failures.
  • Hybrid designs combine both, sometimes called lambda or kappa architectures.

Common technologies include message queues and streaming platforms such as Apache Kafka, processing engines such as Apache Spark and Apache Flink, orchestrators such as Apache Airflow, and cloud managed services. See distributed computing and parallel computing.

Pipelines for machine learning

Machine learning adds its own stages to the pipeline.

  1. Data ingestion and validation. Collect data and check for schema changes, missing values and unusual distributions.
  2. Preparation and feature engineering. Build the features the model needs; see feature engineering. A feature store can keep the same features available to training and to live predictions.
  3. Training. Run model training with tracked settings and data versions.
  4. Evaluation. Test on held-out data and compare with the current model.
  5. Deployment. Package and release the model; see model serving.
  6. Monitoring and retraining. Watch for data drift and trigger retraining.

The practice of automating and managing these stages is called MLOps, and for language models LLMOps; see LLMOps. Sculley and colleagues argued that in real machine learning systems the model is a small part of the code, and that data dependencies and glue code create hidden maintenance costs, which is why well-designed pipelines matter [2].

Qualities of a good pipeline

  • Reliability. Handles failures with retries and alerts.
  • Idempotence. Running a step twice gives the same result, so reruns are safe.
  • Scalability. Copes with growing volume.
  • Data quality checks. Validates data at each stage, stopping bad data from spreading.
  • Lineage and versioning. Records where data came from and which code produced it, for audit and reproduction.
  • Security and privacy. Encryption, access control and handling of personal data in line with laws such as the General Data Protection Regulation.
  • Observability. Metrics and logs that show whether data is fresh and complete; see AI observability.
  • Cost control. Efficient storage and computing.

Common problems

  • Silent failures. Data stops arriving or changes meaning without anyone noticing.
  • Schema changes. An upstream team renames a field and breaks downstream jobs.
  • Training-serving skew. Features computed differently in training and in production.
  • Late or out-of-order data in streams.
  • Pipeline sprawl. Many undocumented jobs that no one owns.

An example pipeline

Consider an online retailer that wants to recommend products. Click and purchase events from the website flow into a streaming platform as they happen. A processing job cleans the events, removes bot traffic and joins them with product and customer tables. The results are written to two places: a data warehouse, where analysts build reports, and a feature store, where the recommendation model reads up-to-date customer features. Each night a batch job retrains the model on the latest data, evaluates it against the current version and, if it performs better, releases it to production. Dashboards track how fresh the data is, how many events were rejected and how the model’s recommendations perform. If the click stream stops or a field changes format, alerts notify the on-call engineer.

Data lakes, warehouses and lakehouses

Pipelines usually end in one of three kinds of store. A data warehouse holds structured, modeled data optimized for analytical queries. A data lake stores raw data of any format, including images, text and logs, cheaply, which suits machine learning. A lakehouse combines the two, adding table management and query performance on top of lake storage. Many organizations use all three, and the pipeline moves data between them, applying quality checks as it goes.

Governance and ownership

As pipelines multiply, governance becomes important. Good practice includes a catalog that lists what data exists and what it means, named owners for each pipeline, service-level agreements for freshness, tests that run automatically when code changes and clear rules for who may see sensitive fields. Treating data products with the same discipline as software, a view sometimes called DataOps, reduces outages and builds trust in the numbers that decisions and models rely on.

FAQs

What is a data pipeline? It is an automated workflow that collects, processes and moves data from sources to destinations such as databases, reports or models.

What is the difference between ETL and ELT? ETL transforms data before loading it into the destination, while ELT loads raw data first and transforms it afterward inside the destination.

What is a machine learning pipeline? It is the chain of steps from data ingestion and preparation through training, evaluation, deployment and monitoring.

What is a streaming data pipeline? One that processes events continuously as they arrive, instead of in scheduled batches.

What tools are used to build data pipelines? Examples include Apache Airflow, Spark, Kafka, dbt and managed cloud services.

More in AI Data and Infrastructure

Key facts

  • ETL stands for extract, transform, load, a pattern in which data is pulled from sources, reshaped and then loaded into a target store such as a data warehouse1.
  • ELT reverses the last two steps, loading raw data into a scalable store first and transforming it there1.
  • Hidden technical debt in machine learning systems often comes from data dependencies and glue code around the model, not the model itself2.

Related concepts

Quick checkWhat is a data pipeline?Show answer

A data pipeline is an automated series of steps that collects data from one or more sources, cleans and transforms it, and moves it to a destination such as a data warehouse, dashboard or machine learning model.

Sources

  1. Ralph Kimball and Joe Caserta, The Data Warehouse ETL Toolkit, Wiley, https://www.wiley.com/en-us/The+Data+Warehouse+ETL+Toolkit-p-9780764567575
  2. D. Sculley et al., Hidden Technical Debt in Machine Learning Systems, Advances in Neural Information Processing Systems, https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue