Distributed computing is the use of multiple computers, linked by a network, that coordinate their work to solve a problem together. Tanenbaum and van Steen describe a distributed system as a collection of independent computers that appears to its users as a single coherent system [1]. The approach lets a task exceed the memory, speed or reliability of any single machine. It underlies web search, cloud services, scientific simulation and, increasingly, artificial intelligence: no single chip can train today’s largest models, so thousands of GPUs are tied together to act as one. It is closely related to parallel computing, with a difference explained below.
Core ideas
- Independent nodes. Each computer has its own processor and memory, and no shared clock.
- Message passing. Nodes communicate by sending messages over a network, which is slower and less reliable than internal memory.
- Concurrency. Many things happen at the same time.
- Partial failure. Parts of the system can fail while others continue, so software must expect and tolerate faults.
- Transparency. Users and programs ideally need not know where work happens.
- Scalability. Adding machines should increase capacity.
Distributed versus parallel computing
The terms overlap. Parallel computing usually means many processing units working simultaneously on one problem, often inside a single machine with shared memory, such as the cores of a GPU. Distributed computing means separate machines connected by a network, each with its own memory. Large AI training combines both: parallelism within each device and distribution across devices.
Why distribute work
- Size. Data or models too large for one machine.
- Speed. Divide work to finish sooner.
- Reliability. Replicate data and services so that failure of one machine does not stop the system.
- Locality. Place computation near users or data, reducing delay.
- Cost. Use many cheap commodity machines instead of one expensive one.
Architectures
- Client-server. Clients request services from central servers.
- Peer-to-peer. Nodes share roles equally, as in file sharing and some blockchains.
- Cluster and cloud. Many machines in a data center managed as a pool; see cloud AI.
- Grid and volunteer computing. Resources from many organizations or volunteers, as in projects that analyze telescope or protein data.
- Edge and fog computing. Computation spread between devices, local gateways and the cloud; see edge AI.
- Microservices. Applications split into small services communicating over a network.
Distributed data processing
In 2004, Dean and Ghemawat described MapReduce, a model in which a job is split into map tasks that run in parallel on pieces of data and reduce tasks that combine the results, with the system handling failures automatically [2]. It inspired Hadoop and later systems such as Apache Spark, Flink and Ray, which are widely used to prepare data for machine learning; see data pipeline and data preprocessing. Distributed file systems and databases store data across many machines with replication.
Distributed training of AI models
Training large neural networks across many accelerators uses several strategies, often combined [3].
- Data parallelism. Every device holds a copy of the model and processes a different slice of each batch. After each step, devices average their gradients, usually with an all-reduce communication operation.
- Tensor (model) parallelism. Individual layers’ large matrices are split across devices, which must exchange partial results at each layer.
- Pipeline parallelism. Groups of layers are placed on different devices, and micro-batches flow through like items on an assembly line.
- Sharding of states. Techniques such as ZeRO and fully sharded data parallelism split optimizer states, gradients and weights across devices to save memory.
- Expert parallelism. For mixture-of-experts models, different experts live on different devices; see mixture of experts.
These depend on very fast interconnects, such as NVLink and InfiniBand, because communication can dominate training time. See AI chip and LLM training.
Federated learning
A different kind of distribution trains models on data that stays on users’ devices or in separate organizations, sharing only model updates; see federated learning. It aims to protect privacy rather than to gain speed.
Challenges
- Network latency and bandwidth. Communication is far slower than computation.
- Failures. In a cluster of thousands of machines, something fails every day, so training jobs save checkpoints and restart.
- Consistency. Keeping replicas in agreement. The CAP theorem states that a system cannot guarantee consistency, availability and partition tolerance all at once.
- Synchronization and stragglers. The slowest node can hold everyone up.
- Complexity. Debugging and testing are harder; see AI observability.
- Security. More machines and links mean more attack surface.
- Energy and cost. Large clusters use great amounts of power.
Tools
Common tools include Kubernetes for orchestration, Apache Spark and Ray for data and machine learning workloads, PyTorch Distributed, DeepSpeed and Megatron-LM for training [3], message brokers such as Kafka, and consensus systems such as etcd and ZooKeeper.
Fault tolerance in practice
When thousands of machines run for weeks, failures are certain. A single faulty GPU, a loose network cable or a software bug can stop a synchronized training job. Systems cope in several ways. They save checkpoints, snapshots of the model and optimizer state, every few minutes or hours, so a job can restart from the last one. They monitor hardware health and swap out bad nodes automatically. They keep spare machines on standby. Some systems use elastic training that continues with fewer devices when one drops out. The cost of a failure is the work done since the last checkpoint, so teams balance the overhead of frequent saving against the risk of lost progress.
In data processing and serving, replication and retries play the same role. Data is copied to several machines, requests are routed to healthy replicas, and operations are designed to be idempotent, so repeating them after a failure does no harm.
Consistency and consensus
A central difficulty is getting separate machines to agree. Without a shared clock, nodes cannot easily tell the order of events. Algorithms such as Paxos and Raft let a group of nodes agree on a sequence of decisions even when some fail, and they power configuration stores and databases. Designers choose between strong consistency, where every read sees the latest write, and eventual consistency, where replicas converge over time, trading correctness guarantees against speed and availability. AI systems mostly use strong guarantees for coordination data and relaxed ones for bulk data such as logs.
FAQs
What is distributed computing? It is using several networked computers that coordinate to solve a problem together.
What is the difference between distributed and parallel computing? Parallel computing uses many processors, often with shared memory, while distributed computing uses separate machines connected by a network.
Why does AI use distributed computing? Large models and data sets exceed a single machine’s memory and speed, so training is spread across many accelerators.
What is data parallelism? Each device holds a full copy of the model and trains on a different part of the data, then they combine updates.
What is MapReduce? A programming model for processing large data sets by running map tasks in parallel and combining results with reduce tasks.
More in AI Data and Infrastructure
Key facts
- A distributed system is a collection of independent computers that appears to its users as a single coherent system1.
- MapReduce, described by Google engineers in 2004, popularized a simple model for processing huge data sets across thousands of machines2.
- Large neural networks are trained by splitting data, model layers or tensors across many accelerators, using data, pipeline and tensor parallelism3.
Related concepts
Quick checkWhat is distributed computing?Show answer
Distributed computing is the use of multiple computers connected by a network that coordinate their work, passing messages to one another, to solve a problem together, often appearing to users as a single system.
Sources
- Andrew Tanenbaum and Maarten van Steen, Distributed Systems: Principles and Paradigms, Pearson, https://www.distributed-systems.net/
- Jeffrey Dean and Sanjay Ghemawat, MapReduce: Simplified Data Processing on Large Clusters, Communications of the ACM, https://doi.org/10.1145/1327452.1327492
- Mohammad Shoeybi et al., Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, https://arxiv.org/abs/1909.08053
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
