A GPU (graphics processing unit) is a processor with thousands of small cores built to perform many calculations at once. It was invented to draw images on screens, where the same operation must be applied to millions of pixels. The same design turned out to suit the heavy arithmetic of neural networks, and since about 2012 the GPU has been the workhorse of modern artificial intelligence. Almost every large neural network today is trained and served on GPUs or on related accelerators such as TPUs and other AI chips.
How a GPU differs from a CPU
A central processing unit (CPU) has a few powerful cores, each very good at running one complicated sequence of instructions quickly, with large caches and sophisticated control logic. It is built for low latency on varied tasks. A GPU takes the opposite approach: it has thousands of simpler cores that run the same instruction on many pieces of data simultaneously, trading single-thread speed for enormous throughput [3].
A useful comparison is a few expert chefs versus a huge kitchen line of cooks doing one step each. For jobs that split into many identical parts, such as multiplying large matrices, the GPU is far faster. For jobs full of branching decisions and dependencies, the CPU does better. Most AI systems therefore use a CPU to coordinate and GPUs to compute.
Why GPUs suit AI
Training and running a deep learning model consist largely of matrix and tensor multiplications, repeated billions of times. These operations are highly parallel, which matches the GPU design. Other helpful features include:
- High memory bandwidth. Fast memory, such as high-bandwidth memory (HBM), keeps the cores supplied with data.
- Tensor cores. Specialized units that multiply small matrices in low-precision formats such as 16-bit and 8-bit numbers, greatly speeding up neural network work.
- Fast interconnects. Links such as NVLink let many GPUs share data, so models can be spread over many chips.
- Mature software. Libraries for deep learning take care of the low-level details.
History
GPUs began in the 1990s as chips for 3D video games. In the early 2000s, researchers noticed that the hardware could be used for general calculations, a practice called GPGPU. NVIDIA released CUDA in 2006, a programming platform that made this practical [2]. In 2012, Krizhevsky, Sutskever and Hinton trained the AlexNet network on two consumer GPUs and won the ImageNet challenge by a large margin, which convinced the field that GPU-trained deep networks were the way forward [1]. Since then, GPUs designed specifically for data centers have grown in power and cost, and demand for them has become a major factor in the AI industry.
Software ecosystem
- CUDA. NVIDIA’s platform, with libraries for neural networks and linear algebra; the most widely used.
- ROCm. AMD’s open software stack.
- OpenCL, Vulkan and Metal. Cross-vendor and platform-specific interfaces.
- Frameworks. PyTorch, TensorFlow and JAX hide GPU details and let developers write models in high-level code.
The maturity of software is a key reason NVIDIA holds a large share of the AI market, alongside competitors such as AMD and Intel, and cloud providers’ own chips.
Using many GPUs
Large models do not fit or train quickly on a single GPU, so work is divided across many. Data parallelism gives each GPU a copy of the model and a different slice of the data. Model parallelism splits the model itself across GPUs. Pipeline parallelism places successive layers on different devices. Large training runs use thousands of GPUs in clusters with high-speed networking; see distributed computing and parallel computing.
Training and inference
Training needs the most computing power and memory, since it processes huge data sets and stores intermediate values. Inference, which is using a trained model, needs less per request but runs at scale, and is often optimized with quantization and model compression to cut memory and cost; see model serving and inference engines.
Limitations and concerns
- Cost and supply. High-end AI GPUs are expensive and have often been in short supply.
- Energy and heat. Data-center GPUs draw hundreds of watts each, requiring strong cooling and raising environmental concerns.
- Memory limits. Model size is often limited by GPU memory.
- Concentration. A few manufacturers and fabrication sites supply most AI chips, creating geopolitical and supply-chain risks, including export controls on advanced chips.
- Alternatives. Specialized chips can be more efficient for some tasks; see edge AI for low-power devices.
Precision, memory and performance
GPU speed for AI depends heavily on the number format. Older scientific computing used 32-bit or 64-bit floating-point numbers. Neural networks tolerate less precision, so modern GPUs offer 16-bit formats such as FP16 and bfloat16, and 8-bit and even 4-bit formats for inference. Using fewer bits means each number takes less memory, more of them move per second and tensor cores can process more per cycle, giving large speedups with little loss in accuracy when done carefully. Training often uses mixed precision, keeping a high-precision master copy of weights while computing in lower precision.
For many workloads, the limiting factor is not arithmetic but moving data. A large language model must read all its weights from memory to produce each word, so memory bandwidth often sets the speed of text generation. This is why high-bandwidth memory and techniques that reduce memory traffic, such as caching and batching requests, matter as much as raw compute. Practitioners summarize this with the idea of being compute-bound or memory-bound.
Beyond data centers
GPUs appear at every scale. Laptops and phones contain small integrated GPUs that run image effects and on-device models. Workstations with a single powerful card are common for research and for running open models locally. Cloud providers rent GPUs by the hour, which lets small teams use hardware they could not buy; see cloud AI. Embedded GPU modules power robots, drones and vehicles that must process sensor data on board, linking to autonomous robots.
Outside AI, GPUs are used for video games, film rendering, cryptocurrency mining, weather and physics simulation, drug discovery and video editing. Demand from these fields has at times competed with AI for supply.
FAQs
What is a GPU? It is a processor with thousands of small cores that performs many calculations in parallel, originally for graphics and now widely used for AI.
Why are GPUs used for AI? Neural networks rely on large matrix multiplications that can be run in parallel, which GPUs do very quickly.
What is the difference between a CPU and a GPU? A CPU has few powerful cores for varied tasks, while a GPU has thousands of simpler cores for parallel work.
What is CUDA? It is NVIDIA’s platform for programming GPUs for general computing, released in 2006.
Can AI run without a GPU? Yes, small models run on CPUs, but large models are far slower without GPUs or other accelerators.
Key facts
- The 2012 AlexNet network, trained on two consumer GPUs, won the ImageNet challenge by a wide margin and showed the power of GPU-trained deep learning1.
- NVIDIA introduced the CUDA platform in 2006, allowing developers to program GPUs for general-purpose computing2.
- GPUs excel at matrix multiplication, the core operation of neural networks, because they run thousands of threads at once3.
Related concepts
Quick checkWhat is a GPU?Show answer
A GPU, or graphics processing unit, is a processor with thousands of small cores built to perform many calculations at the same time. Originally made for rendering images, it is now the main hardware for training and running AI models.
Sources
- Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, ImageNet Classification with Deep Convolutional Neural Networks, Advances in Neural Information Processing Systems, https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html
- John Nickolls et al., Scalable Parallel Programming with CUDA, ACM Queue, https://doi.org/10.1145/1365490.1365500
- John Hennessy and David Patterson, Computer Architecture: A Quantitative Approach, Morgan Kaufmann, https://www.elsevier.com/books/computer-architecture/hennessy/978-0-12-811905-1
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
