NVIDIA H200 (Hopper): The Complete Guide

Codename GH100 (same die family as H100), architecture Hopper. General availability followed in Q2 2024 from system manufacturers and cloud providers 1. Ships as SXM, the air-cooled H200 NVL PCIe variant, and inside the GH200 Grace Hopper Superchip 1.

NVIDIA’s own press release states H200 delivers “nearly doubling inference speed on Llama 2, a 70 billion-parameter LLM,” compared with H100 1 — a company-published, model-specific performance claim, not an independent benchmark. The release lists ASRock Rack, ASUS, Dell, Eviden, GIGABYTE, HPE, Ingrasys, Lenovo, QCT, Supermicro, Wistron and Wiwynn as system partners, and AWS, Google Cloud, Microsoft Azure, Oracle Cloud Infrastructure, CoreWeave, Lambda and Vultr as cloud partners 1.

H200’s architectural features are the Hopper features already documented for H100: Transformer Engine, fourth-generation NVLink, second-generation Multi-Instance GPU, DPX instructions for dynamic programming, and built-in confidential computing 2. NVIDIA’s H200 materials do not describe any new compute-side architectural feature — the announcement is framed entirely around the memory upgrade 1.

Figures per NVIDIA’s own H200 product page 3. Every Tensor Core FLOPS figure here (FP64, FP64 Tensor, FP32, TF32, BF16/FP16, FP8, INT8) is numerically identical to H100 SXM5’s published figures — see the “Hidden in plain sight” section.

Because H200 is a memory-subsystem revision of the H100 GH100 die, its foundry and process are identical to H100’s: TSMC, custom 4N 2. Neither NVIDIA nor TSMC publishes individual fab locations for either chip.

NVIDIA’s own release names its system partners as ASRock Rack, ASUS, Dell, Eviden, GIGABYTE, HPE, Ingrasys, Lenovo, QCT, Supermicro, Wistron and Wiwynn, and its cloud partners as AWS, Google Cloud, Microsoft Azure, Oracle Cloud Infrastructure, CoreWeave, Lambda and Vultr 1. The GH200 Grace Hopper Superchip pairs a Grace CPU with H200-class HBM3e memory in a single package 1.

This is one of the few chips on this site where a cloud partner’s own published rate card gives a specific, checkable number rather than a general “contact sales” page — CoreWeave’s pricing page states these figures directly 4. AWS, Azure and Google Cloud also offer H200 instances but their exact rates were not independently re-verified for this page; check each provider’s own pricing page for current figures, since cloud rates change.

Because H200’s compute Tensor-core throughput figures are identical to H100’s, its real-world performance gains come almost entirely from serving larger models and longer contexts without running out of memory, and from the higher memory bandwidth feeding the unchanged compute units faster. NVIDIA’s Llama-2-70B inference claim 1 is consistent with a memory-bandwidth-bound workload benefiting from the 43% bandwidth increase (3.35 TB/s to 4.8 TB/s).

H200 is not a new architecture generation — it is a memory-upgraded H100. The next architectural generation NVIDIA announced is Blackwell, with the B200 GPU revealed at GTC in March 2024 1 (context from the same period; see the B200 page for its own primary sources).

Comparing NVIDIA’s own H100 and H200 specification pages side by side 23: FP64 34/67 TFLOPS, FP32 67 TFLOPS, TF32 989 TFLOPS (sparse), BF16/FP16 1,979 TFLOPS (sparse), FP8/INT8 3,958 TFLOPS/TOPS (sparse), and 900 GB/s NVLink are identical across both chips. NVIDIA did not change the compute die at all between H100 and H200 — it changed only the memory type (HBM3 to HBM3e), capacity (80GB to 141GB) and bandwidth (3.35 TB/s to 4.8 TB/s).