Why AI runs on special chips, what is inside one, why memory matters more than speed, how thousands of chips are joined into one computer, and why they need so much power. Start simple, go as deep as you like.
Stuck at any point? Ask an AI about this page →An ordinary computer processor (a CPU) is like a few very clever workers who handle one job after another. An AI chip is like thousands of workers who each do one small sum at the same moment.
AI models are built from huge grids of numbers, and running them means multiplying those grids together billions of times. NVIDIA's own guide calls these matrix multiplications "a fundamental building block" of neural networks 1. Thousands of workers doing sums at once is exactly what that needs.
The difference is scale. The biggest server CPUs today have 192 cores (AMD's EPYC 9965) or 128 cores (Intel's Xeon 6900P) 2 3. An NVIDIA H100 GPU has 132 "streaming multiprocessors" holding 16,896 basic cores and 528 Tensor Cores built specifically for matrix maths 4. A CPU is built to be quick at anything; a GPU is built to do the same simple thing thousands of times in parallel 5.
Most of a neural network's work reduces to general matrix multiplies (GEMMs): fully connected layers, attention and convolutions can all be expressed that way 1. The question for any chip is whether a given operation is limited by maths or by memory. NVIDIA frames this as arithmetic intensity, the number of calculations per byte moved: if it is lower than the chip's ratio of maths speed to memory speed, the chip waits on memory 5. Matrix-vector products, common when an AI generates text one word at a time, are always memory-limited 1. That single idea explains most AI chip design choices.
Inside an AI chip are two kinds of worker: general cores that can do any sum, and special matrix engines that only multiply grids of numbers but do it extremely fast. NVIDIA calls its matrix engines Tensor Cores; Google calls its version the MXU 6 7.
NVIDIA introduced Tensor Cores with its Volta chips; each one multiplies small blocks of numbers and adds the result in one step, and can add at higher precision than it multiplies 5. A Blackwell Ultra GPU has 160 multiprocessors and 640 fifth-generation Tensor Cores 8.
Google took a different route with its TPU. Each TPU core has matrix units built as a systolic array: a grid of 128 by 128 multiply-accumulate cells (256 by 256 on newer TPUs) where numbers flow through like a wave, each cell passing its result to the next 7. Google's own documentation says TPUs suit work dominated by matrix maths and are unsuitable for programs full of branches 9.
The original TPU, described by Google engineers in 2017, had a single matrix unit of 65,536 8-bit multiply-accumulators delivering 92 trillion operations per second, with 28 MiB of on-chip memory; it had been in Google's data centres since 2015 and was on average 15 to 30 times faster than the CPUs and GPUs of its time on Google's inference work, with 30 to 80 times better performance per watt 10. The design lesson has held: give up flexibility, spend the silicon on matrix maths and on-chip memory, and efficiency rises sharply.
Computers store numbers with a fixed number of digits. AI does not need many, so chips use shorter numbers: 16, 8 or even 4 bits instead of 32. Shorter numbers mean the chip can do more sums at once and fit bigger models in memory. That is why chip makers quote their biggest speeds at "FP4" or "FP8".
At 4 bits, the trick is shared scale factors. The industry's open MX standard, contributed in 2023 by AMD, Arm, Intel, Meta, Microsoft, NVIDIA and Qualcomm, groups 32 numbers that share one power-of-two scale, and defines MXFP8, MXFP6, MXFP4 and MXINT8 13. NVIDIA's NVFP4 uses smaller blocks of 16 values with a more precise FP8 scale, plus one scale per tensor 12. When comparing chips, check that the precision matches: an FP4 figure is roughly double an FP8 figure on the same hardware.
A chip can only work as fast as it can be fed. AI chips use a special memory called HBM (high-bandwidth memory): memory chips stacked on top of each other and placed right next to the processor, so data has a very short distance to travel.
The industry standard for the newest version, HBM4, was published by JEDEC in April 2025. Each stack has a 2,048-bit connection running at up to 8 Gb/s per pin, for up to 2 terabytes per second, and can be 4, 8, 12 or 16 chips high with up to 64 GB 14. A Blackwell Ultra GPU carries eight 12-high stacks, 288 GB in total, at 8 TB/s 8; NVIDIA's Rubin GPU moves to HBM4 with up to 288 GB and up to 22 TB/s 15.
Researchers call the gap between compute and memory the "memory wall". Over 20 years, peak AI hardware compute grew about 60,000 times while memory bandwidth grew about 100 times and chip-to-chip bandwidth about 30 times; server compute scales about 3.0 times every two years against 1.6 for memory 16. The authors conclude memory is now the main bottleneck, especially for serving models to users 16. This is why HBM supply, not chip design, often decides how many AI chips can be built.
The biggest AI models do not fit on one chip, so chips are joined with extremely fast cables to act as one. NVIDIA's GB200 NVL72 joins 72 GPUs and 36 processors in one rack, with each GPU able to talk to the others at 1.8 terabytes per second 17.
There are two kinds of link. Scale-up joins chips inside a rack so they share memory almost like one chip; scale-out joins racks across a building. NVLink is NVIDIA's scale-up link, with NVLink Switch chips giving every GPU a full-speed path to every other 18. Google joins its TPUs with its own inter-chip network into "pods" 7.
Open alternatives now exist. The UALink Consortium published its 1.0 scale-up specification in April 2025: 200 Gb/s per lane, up to 1,024 accelerators in a pod, with more than 85 member companies including AMD, Apple, AWS, Google, Intel, Meta and Microsoft 19. The Ultra Ethernet Consortium published its 1.0 scale-out specification in June 2025, designed to reach millions of endpoints over Ethernet 20. NVIDIA's figures for its sixth-generation NVLink differ between pages: its NVLink page gives 3 TB/s per GPU and 216 TB/s per Vera Rubin NVL72 rack 18, while its Rubin technical blog gives 3.6 TB/s and 260 TB/s 15.
Training is teaching: the model looks at huge amounts of data and adjusts itself. Inference is using: the finished model answers your question 21. Training happens in giant bursts; inference happens every time anyone uses the AI.
Chips are now being designed for one or the other. Google says Ironwood, its seventh-generation TPU announced in April 2025, is the first TPU designed specifically for inference, with 192 GB of HBM per chip and pods of up to 9,216 chips 22. Groq's LPU was built only for inference and keeps its data in on-chip memory rather than HBM 23.
Training compute keeps climbing. The research group Epoch AI estimates the compute used to train frontier models grew about 4 to 5 times a year from 2010 to 2024 24, and counted more than 30 models from 12 developers trained with over 10^25 operations by mid-2025 25. These are research estimates, not company disclosures. Inference, meanwhile, favours memory bandwidth and low precision, which is why inference chips lean on FP4 and on-chip memory 21 23.
Cerebras's WSE-3 packs 4 trillion transistors and 900,000 AI cores onto one wafer, with 44 GB of on-chip memory, made on TSMC's 5 nm process 26. Groq says its LPU reads on-chip memory at over 80 TB/s, against about 8 TB/s for a GPU's off-chip HBM, and runs programs the same way every time because the compiler plans every step in advance 23.
On devices: Apple's M4 Neural Engine does up to 38 trillion operations per second 27; Apple's M5 adds a neural accelerator to every GPU core and claims over four times the peak GPU AI compute of M4, with 153 GB/s of memory bandwidth 28. Qualcomm's Snapdragon X2 Elite laptop chip has an 80 TOPS NPU 29. Cerebras's CS-4 system, announced in August 2026, uses a "WSE-3 Turbo" at 250 petaflops per wafer by the company's own figures 30.
AI chips use a lot of electricity. One NVIDIA GPU can draw up to 1,400 watts 8. A full rack of 72 draws around 120 kilowatts 31, so racks are now cooled with liquid instead of fans.
Power per chip keeps rising: NVIDIA's maximum GPU power went from 700 W (Hopper) to 1,200 W (Blackwell) to 1,400 W (Blackwell Ultra) 8. Air cooling reaches its limits as racks go past about 20 kW and approach 50 kW, according to cooling maker Vertiv 32. The GB200 NVL72 uses cold plates on each processor with coolant piped through the rack 31, and NVIDIA describes Vera Rubin NVL72 as fully liquid cooled 15.
The International Energy Agency estimates data centres used about 415 TWh of electricity in 2024, around 1.5% of the world's total, and projects this to more than double to about 945 TWh by 2030, with AI the biggest driver; the US was 45% of data-centre use, China 25% and Europe 15% 33. Efficiency per chip keeps improving (Google says Ironwood has twice the performance per watt of its previous TPU 22), but total demand is growing faster than efficiency.
33 sources, all checked September 2026. Official = the company or organisation’s own page; Filing = a document filed with a regulator or an annual report; Standard = a published industry standard; Paper = a peer-reviewed or conference paper; Research = an independent research body; Government = a government page; Press = news coverage, used only where no official source exists.
Opens your assistant with this page as the source, and a question rather than a summary. It will ask what you are building before it answers.
Nothing is sent from here. The link carries only this page’s title and address.