Specialty Digest

Global English editionLearners’ Study Dashboard — in development
The Knowledge Library for All

TPU

nounAI Data and Infrastructurealso tensor processing unit, Google TPU, TPU pod

In one line

A TPU, or tensor processing unit, is a custom chip designed by Google to accelerate the matrix calculations at the heart of neural networks, used in its data centers and offered through its cloud.

A TPU (tensor processing unit) is a custom chip designed by Google to speed up the calculations used in neural networks. Unlike a general-purpose processor, it is an application-specific integrated circuit (ASIC) built around one job: multiplying large arrays of numbers, called tensors. Google began running TPUs in its data centers in 2015, revealed them publicly in 2016 and described the first generation in detail in a 2017 paper [1]. TPUs power Google services such as Search, Translation and Photos, and are rented through Google Cloud. They are one family of AI chips that compete with the GPU.

Why Google built it

In about 2013, Google engineers estimated that if users spoke to their phones for just a few minutes a day using voice search powered by neural networks, the company would need to double its data center capacity. Instead of buying that much general hardware, Google designed a chip focused on neural network inference. The result, described by Jouppi and colleagues, ran inference 15 to 30 times faster than contemporary CPUs and GPUs in Google’s tests and delivered 30 to 80 times better performance per watt [1].

How a TPU works

The heart of a TPU is the matrix multiply unit, a large grid of arithmetic units arranged as a systolic array. In a systolic array, data flows rhythmically from one unit to its neighbors, with each unit multiplying and adding as the values pass through. Because values are reused as they move, the chip needs far fewer reads from memory, which saves time and energy. The first-generation TPU held 65,536 8-bit multiply-accumulate units in a 256 by 256 grid [1].

Other features include:

  • Reduced precision. Early TPUs used 8-bit integers for inference. Later versions added bfloat16, a 16-bit format that keeps the range of 32-bit numbers and became widely used in training.
  • On-chip memory. Large buffers keep data close to the arithmetic units.
  • High-bandwidth memory. Later generations pair the chip with fast stacked memory.
  • Simple control. The chip is driven by the host computer and does not need the complex machinery of a CPU.

TPUs are designed for the dense linear algebra of deep learning, including transformers, and are less suited to irregular, branching code.

Generations

  • TPU v1 (2015). Inference only, 8-bit.
  • TPU v2 and v3. Added floating-point math and training, and were connected into larger systems called pods with fast links.
  • TPU v4. Introduced optical switching so that a large pod could be reconfigured, and was reported to connect 4,096 chips in one system [2].
  • Later generations. Google has continued to release newer TPUs for training and inference, with more memory, speed and efficiency. Specifications change often, so consult Google’s current documentation.

TPU pods and scaling

Training a large model needs many chips working together. TPUs are linked in pods through a dedicated high-speed network arranged in a grid or torus, so that thousands of chips can behave like one machine. Software splits the model and data across the chips; see distributed computing and parallel computing. Google has used TPU pods to train very large language models, and it offers them as a service through cloud AI.

Software

TPUs are programmed mainly through frameworks that compile models to the chip. TensorFlow was the original path, and JAX has become popular for research, with PyTorch also supported through compatibility layers. The compiler XLA turns a model’s operations into efficient TPU instructions. This compile step lets the system fuse operations and plan memory use, though it can make debugging different from working with GPUs.

TPU versus GPU

  • Specialization. A TPU is built for tensor math, while a GPU keeps general graphics and compute capabilities, giving more flexibility.
  • Availability. TPUs are used inside Google and rented through Google Cloud. GPUs are sold by several vendors and run in all major clouds and on personal machines.
  • Ecosystem. The GPU software ecosystem, especially CUDA, is broader; TPU strengths are tight integration with Google’s stack.
  • Efficiency. For well-matched workloads, TPUs can offer strong performance per watt and per dollar, though results depend on the model and the benchmark.

Other companies have also built their own accelerators; see AI chip.

Uses

TPUs serve search ranking, translation, speech recognition, photo search, recommendation, protein structure prediction and training of large language models. Small versions known as Edge TPUs were made for low-power devices; see edge AI.

Design trade-offs

Specializing hardware brings clear gains but also risks. A chip takes years to design, while neural network architectures change quickly. The designers of the first TPU bet that matrix multiplication would stay central, and the bet paid off as transformer models are dominated by exactly that operation. Even so, new needs keep appearing, such as handling the very large embedding tables used in recommendation, or the memory-heavy attention step in long-context language models. Later TPUs added dedicated units and larger memory to address them.

Another trade-off is flexibility versus efficiency. A fixed-function design wastes little silicon on features that AI does not use, so more of the chip does useful work and less energy is spent moving data. The cost is that unusual operations may run slowly or require the compiler to rewrite them. Developers evaluating hardware therefore benchmark their own models, instead of relying on peak numbers, because real speed depends on batch size, model shape and software quality; see AI benchmark.

Energy and sustainability

Power use is a major concern in AI. Google has reported steady gains in efficiency across TPU generations, and TPUs are typically deployed in liquid-cooled data center racks. Efficiency matters both for cost and for the environmental footprint of training and serving large models, a topic within responsible AI.

Impact on the AI industry

TPUs showed that a company could gain an edge by designing its own hardware around its own workloads. Since their introduction, many cloud and technology companies have built custom accelerators, and chip startups have proposed other designs, from wafer-scale processors to chips built around on-chip memory. The competition has pushed down the cost of training and serving models and reduced dependence on a single supplier. For researchers, access to TPUs through academic programs and notebook services has allowed experiments at a scale that would otherwise require a large budget.

FAQs

What is a TPU? It is a Google-designed chip specialized for the matrix calculations in neural networks.

What does TPU stand for? Tensor processing unit.

Is a TPU better than a GPU? It depends. TPUs can be very efficient for matching neural network workloads, while GPUs are more flexible and widely available.

Can I use a TPU? Yes, through Google Cloud and some notebook services, using frameworks such as JAX and TensorFlow.

Who makes TPUs? Google designs them, and they are manufactured by chip fabrication partners.

More in AI Data and Infrastructure

Key facts

  • Google began deploying TPUs in its data centers in 2015 and described the first generation in a 2017 paper1.
  • The first-generation TPU contained a matrix multiply unit of 65,536 8-bit multiply-accumulate units1.
  • Google reported that its first TPU ran neural network inference 15 to 30 times faster than contemporary CPUs and GPUs in its tests, with much better performance per watt1.

Related concepts

Quick checkWhat is a TPU?Show answer

A TPU, or tensor processing unit, is a custom chip designed by Google to accelerate the matrix and tensor calculations used in neural networks, offered inside Google data centers and through Google Cloud.

Sources

  1. Norman Jouppi et al., In-Datacenter Performance Analysis of a Tensor Processing Unit, Proceedings of the 44th International Symposium on Computer Architecture, https://arxiv.org/abs/1704.04760
  2. Norman Jouppi et al., TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings, https://arxiv.org/abs/2304.01433

Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue