Amazon Inferentia: The Complete Guide

At announcement, AWS gave almost no technical detail: then-AWS-CEO Andy Jassy described only that Inferentia chips would deliver “hundreds of TOPS (tera operations per second) of inference throughput” each, with “multiple AWS Inferentia chips” usable together for thousands of TOPS 1. AWS filled in the precise figures only later, on its own product page: “up to 128 tera operations per second (TOPS) of performance” per chip 3 — this page flags that the 2018 announcement and the current product page state two different levels of precision, both from AWS itself, rather than treating either alone as the full picture.

Inferentia was designed by Annapurna Labs, the Israeli chip company Amazon bought in 2015 4. Its first major public customer was Amazon’s own Alexa: by November 2020, AWS said the majority of Alexa’s workloads had moved to Inf1 instances, citing “25% lower end-to-end latency, and 30% lower cost compared to GPU-based instances for Alexa’s text-to-speech workloads” 5 — an early, concrete proof point for a chip that launched with almost no published detail.

AWS’s own description: “The first-generation Inferentia has 8 GB of DDR4 memory per chip and also features a large amount of on-chip memory” 3 — AWS does not quantify that on-chip figure precisely. Data types supported: FP16, BF16 and INT8 3. AWS has never published a process node or power figure for Inferentia, and this page found no independent teardown or press report naming either — unlike later chips, where press has at least named a likely foundry (see the Trainium2 and Trainium3 pages on this site).

Instance table per AWS’s own December 2019 launch post 2, unchanged on the current instance-types page 6.

Unlike Trainium2 and Trainium3, where press reports (not Amazon) name TSMC, this page found no report of any kind — press, analyst or teardown — naming a foundry for the original Inferentia. Coverage of the 2018 announcement noted explicitly that Amazon gave almost no technical detail at all: SiliconANGLE’s launch-day report says then-AWS-CEO Andy Jassy “provided few details on the design or specifications” 7, and Data Center Knowledge’s coverage the next day says he “did not share any details about the chip’s design or performance” 8. This page treats the foundry as genuinely unpublished rather than guessing from later Amazon chips.

Beyond Amazon’s own Alexa, AWS has published named customer results for Inf1: Sprinklr says it was “able to significantly improve the performance of one of our NLP models and improve the performance of one of our computer vision models” 9; Finch Computing reported inference costs down “over 80%” versus GPUs; Dataminr reported “up to 9x better throughput per dollar”; Autodesk reported “4.9x higher throughput” over a GPU instance for its NLU models; Condé Nast reported a “72% reduction in cost”; and Amazon’s own Search team reported inference costs down 85% 6. All are AWS-published customer quotes, not independently audited figures — Sprinklr’s own testimonial does not give a specific percentage, unlike the others.

Prices shown are AWS’s US East (default region) list prices as published on its Inf1 instance-types page; AWS shows a region selector and prices vary by region 6. There is no separate per-chip price — Inferentia has never been sold as a standalone part, only rented as part of an instance.

These are two different AWS-published figures from two different dates, not one consistent claim — this page states both rather than picking one, since AWS itself has not explained the change (a different GPU comparator, updated pricing, or simply a revised marketing figure are all plausible, unconfirmed explanations). Neither is an independent third-party benchmark; both are AWS’s own comparisons against its own GPU instances.

Amazon’s AI-silicon roadmap split into two families after this chip: Inferentia stayed inference-only through Inferentia2, while Trainium (announced 2020, covered on its own page on this site) took on training. In practice, Amazon’s newer Trainium chips now handle most inference too, according to the brand-level roadmap on this site 10 — Inferentia and Inferentia2 remain the dedicated inference-only line.

Contemporary coverage of the 28 November 2018 announcement is notably thin on Amazon’s own words: reports from the event describe then-AWS-CEO Andy Jassy giving “few details on the design or specifications” 7 and not sharing “any details about the chip’s design or performance” 8. The precise 128-TOPS figure, the 8GB DDR4 memory figure, and the four-NeuronCore count all appear only later, on AWS’s own product page, not in the launch announcement itself 1 3.