HBM: the AI memory bottleneck
HBM stacks DRAM vertically for bandwidth; HBM4 moves over 2TB/s per stack
The three lines
- HBM stacks 4 to 16 DRAM dies connected by through-silicon vias to maximise bandwidth
- The HBM4 standard set in 2025 specifies a 2,048-bit interface and over 2TB/s per stack
- By Q2 2025 shipments, SK hynix held 62%, Micron 21% and Samsung 17% of the market
Key questions
- What is HBM memory
- High Bandwidth Memory: DRAM dies stacked vertically and connected by through-silicon vias, sitting beside the processor on a shared interposer. The short, very wide connection gives far more bandwidth than conventional memory on a motherboard.
- Why do AI chips need HBM
- A large language model reads its entire parameter set from memory for every token it generates. Accelerator throughput is bounded by how fast it can be fed, not by arithmetic capability, so bandwidth decides usable performance.
- Who makes HBM
- Three suppliers. On Q2 2025 shipment share Counterpoint put SK hynix at 62%, Micron at 21% and Samsung at 17%. No fourth entrant is close to qualification.
Every discussion of AI hardware eventually arrives at memory, usually without explaining why. This is a reference piece on HBM — what it is, why it became the constraint on accelerator performance, and who controls the supply. It is written to be linked to rather than read once.
1. What HBM is
Conventional computer memory sits in modules some distance from the processor, connected by a relatively narrow bus. That arrangement is flexible and cheap, and its bandwidth ceiling is low.
High Bandwidth Memory takes the opposite approach. Between four and sixteen DRAM dies are stacked vertically and connected by through-silicon vias — copper channels drilled straight through each die rather than routed around its edge. The stack sits immediately beside the processor on a shared interposer.
Two things follow. The connection is extremely wide, because thousands of vertical channels can run through a stack where only a few hundred traces would fit on a board. And it is very short, which cuts the energy cost per bit moved. Bandwidth rises by an order of magnitude; packaging complexity and cost rise with it.
2. Why it became the bottleneck
An accelerator's usable throughput is set by how fast it can be fed, not by how fast it can multiply. A large language model reads its entire parameter set from memory for each token it produces. Arithmetic units sit idle waiting for data.
This is why generational HBM specifications track AI capability more closely than transistor counts do.
| Generation | Interface | Bandwidth per stack | Status |
|---|---|---|---|
| HBM2E | 1,024-bit | about 460GB/s | legacy |
| HBM3 | 1,024-bit | about 819GB/s | shipping |
| HBM3E | 1,024-bit | about 1.2TB/s | volume, 12-high 36GB |
| HBM4 | 2,048-bit | over 2TB/s | in production, up to 16-high 64GB in spec |
The HBM4 step is structural rather than incremental. Doubling the interface width to 2,048 bits roughly doubles bandwidth without forcing per-pin speeds into a region where signal integrity and power degrade sharply. Its base die also moves from a memory process to a foundry logic process, which makes memory suppliers dependent on foundry partners for a component they used to make themselves.
3. Who controls supply
Three companies make HBM at volume. On Q2 2025 shipment share, Counterpoint put SK hynix at 62%, Micron at 21% and Samsung at 17%. No fourth supplier is near qualification, and the barrier is not lithography but packaging yield — stacking sixteen dies with thousands of vias each, without a single fatal defect, is a manufacturing problem few can solve economically.
That concentration is the strategic fact. When one supplier holds roughly three-fifths of a component that gates AI accelerator performance, its capacity decisions propagate into the availability and price of AI compute worldwide.
Two caveats on the numbers. Shipment-share estimates come from a single research firm and differ from revenue-based rankings, since generation mix affects average selling price. And no supplier publishes HBM pricing, so any cost comparison across generations rests on market research rather than disclosure.