What an AI inference chip is — why companies build separate silicon from training GPUs
An AI inference chip is built to run a finished model rather than to create one. Training happens a few times per model and needs flexibility; inference happens on every user request, so throughput, latency and efficiency per watt decide everything, because electricity becomes a recurring unit cost. That is why companies that buy GPUs for training increasingly design their own silicon for inference. The cost is generality — a narrow chip runs cheaply but is hard to reuse when model architectures change, and in practice the binding constraint is usually memory bandwidth rather than raw compute
The three lines
- Split — Training builds a model, inference runs it. The frequency and the requirements are fundamentally different
- Metric — What matters in inference is not peak speed but work per watt, because electricity is a recurring cost
- Limit — Narrower means more efficient but less reusable. The bottleneck is usually memory bandwidth
Key questions
- How is an inference chip different from a training GPU?
- **The job has a different shape.** Training adjusts a model's parameters step by step, so it must run both the **forward and backward passes**, and researchers change architectures mid-experiment — which demands **flexibility**. Inference feeds input through fixed parameters to produce output, so it needs only the **forward pass**, and because the architecture no longer changes, the circuitry can be designed for that architecture specifically. That difference splits the design philosophy. A training GPU is built on the premise that anything might arrive; an inference chip is built to do one job as cheaply as possible. The frequency differs too — a model is trained a handful of times, then **runs hundreds of millions of times a day**. This page covered the commercial implications in "Training versus inference: why the same GPU sells twice."
- Why is performance per watt the headline number?
- **Because in inference the electricity bill is the unit cost.** Training spend behaves like development cost, incurred once per model. Inference spend is a **variable cost** incurred on every user query. So the competition is not for the fastest chip but for **how much work the same electricity does**. There is a second reason. A data centre has a hard **ceiling on the power it can draw**. Inside a building with fixed supply, work per watt determines **the total throughput that building can deliver** — buying more chips is pointless if there is no power to plug them into. This page covered the scale of that ceiling in "AI data centre power: why it is counted in gigawatts." When OpenAI unveiled its Jalapeño inference chip in August 2026, the figure it led with was exactly this one: **a 700-watt part beating 1,200- and 1,400-watt alternatives**.
- So do GPUs become unnecessary?
- **No — the work divides.** Three reasons. First, **training remains GPU work**. Large clusters building frontier models need flexibility and a mature software ecosystem, which narrow silicon does not readily replace. Second, **architectures keep changing**. Optimise a chip for today's dominant model shape and it may be obsolete when the shape shifts; if design, validation and volume production take longer than a model generation, the chip arrives late. Third, **software**. However good the silicon, real performance depends on compilers, kernels and framework support — much of Nvidia's durable position sits in that layer rather than in the hardware. Consistent with all three, OpenAI announced its own chip while saying it **would keep buying Nvidia's**. Buying and building run in parallel.
"AI chip" sounds like one category. In practice there are two jobs, and they demand different things.
Building a model (training) and running a finished model (inference).
An inference chip does only the second. What it gives up in exchange is worth understanding.
1. Training versus inference
| Item | Training | Inference |
|---|---|---|
| Job | adjusts parameters | produces answers from fixed parameters |
| Compute | forward + backward pass | forward only |
| Frequency | a few times per model | every request |
| Cost type | closer to development spend | closer to variable cost |
| What matters most | flexibility, mass parallelism | work per watt, latency |
| Architecture changes | often, during research | fixed after deployment |
The final row carries the design consequence. During training you do not know what will arrive; during inference you do.
Knowing what will arrive lets you build circuitry for that and nothing else — and skip the cost a general-purpose processor pays to keep every option open.
2. Why "per watt"
Two reasons the headline metric for inference silicon is work per watt rather than peak speed.
① Electricity is a recurring cost.
Training electricity is spent once, building the model. Inference electricity is spent every time a user asks something. As a service grows, only this line grows with it.
② Data centres have a power ceiling.
A building can draw only so much. Buy more chips and there is nowhere to plug them in. Under that constraint, work per watt sets the total capacity of the building.
| Condition | Work per watt 1.0 | Work per watt 1.7 |
|---|---|---|
| Same power budget | 100 units | 170 units |
| For the same output | 100 power | 59 power |
When OpenAI unveiled Jalapeño on August 25, 2026, the figures it led with aimed squarely at this metric — 1.5 to 1.9 times work per watt, and a 700-watt part outperforming 1,200- and 1,400-watt alternatives. Covered in "OpenAI unveils Jalapeño, its first chip."
3. The real bottleneck is memory, not compute
A common misreading: a slow chip is usually not slow at arithmetic.
Running a large language model requires the chip to fetch model weights from memory at every step. The arithmetic finishes almost instantly; if the fetch cannot keep pace, the compute units idle.
| Situation | What limits it |
|---|---|
| Answering one user quickly | memory bandwidth |
| Processing many requests batched together | shifts toward compute |
So an inference chip largely lives or dies on what memory it has and how wide the path to it is. It is why each HBM generation produces a step change in AI chip performance — and why analysts responded to Jalapeño's benchmark by pointing out that it uses newer HBM4 than the parts it was measured against.
This page covered the underlying structure in "What HBM is" and "HBM4 versus HBM3E."
4. What you gain and lose by narrowing
| Gains | Loses | |
|---|---|---|
| Narrow chip (inference-only) | efficiency per watt, unit cost, latency | generality — useless if the architecture changes |
| Wide chip (general GPU) | runs anything, mature ecosystem | efficiency, power, unit price |
The loss bites through time.
Designing, validating and volume-producing a chip is not quick. If the dominant model architecture shifts in the interim, the optimisations aimed at the old shape stop paying. When model generations turn over faster than chip development cycles, narrow silicon is late at birth.
And then there is software. A good chip with immature compilers, kernels and framework support does not deliver its paper performance. A large share of Nvidia's defensible position sits in that layer, not in the die.
5. What is actually happening — buying and building
The pattern emerging through 2026 is not replacement but division of labour.
| Stage | Who does it |
|---|---|
| Frontier model training | large general-purpose GPU clusters |
| Production inference | increasingly custom silicon |
OpenAI unveiled Jalapeño and said it would keep buying Nvidia chips. Read against the table, that is coherent: keep purchasing for the building, bring the daily repeated cost centre in-house.
The pressure this puts on Nvidia is not lost market share but compressed pricing on the highest-volume slice — what CNBC called a threat to margins.
Nvidia's numbers the same week show the tension intact: second-quarter revenue $96.2 billion, Data Center $89.02 billion at roughly 92 percent of the company, third-quarter guidance $108 billion — a record, and the stock fell after hours.
6. Common questions
Q. Is the AI chip in my phone or laptop an inference chip? Yes. On-device AI silicon (NPUs and similar) runs models; it does not train them. Covered in "What on-device AI is."
Q. If inference chips get cheaper, do AI services get cheaper? Directionally yes, but not automatically. Pricing depends on demand, competition and model size as well as cost. In August 2026 one provider cut output pricing (GPT-5.6 Sol, $30 to $20) while another raised it sharply (DeepSeek, output up to 371 percent) in the same month. Covered in "What AI token pricing is."
Q. Is an inference chip an ASIC? There is a spectrum, depending on how much is fixed. It runs from fully hard-wired designs tuned to a single model to intermediate forms like Jalapeño, presented as a general-purpose LLM inference accelerator. Narrower means more efficient and shorter-lived.
7. What could not be confirmed
- Design trade-offs — general principles, not verification of any product's internals.
- Training-to-inference compute ratio — no reliable source found; no ratio given.
- Per-company silicon — not cross-checked, so no company figures are cited.
- Jalapeño performance — vendor-measured, no independent reproduction.
- Memory bottleneck — widely accepted, but varies with model size and batching.
Sources
- CNBC — OpenAI's Jalapeño AI chip brings new 'threat' to Nvidia margins as custom silicon gains ground
- The Decoder — OpenAI's first custom chip "Jalapeño" reportedly beats Nvidia's Blackwell and Rubin in inference benchmarks
- SemiAnalysis — OpenAI Jalapeño: Better Than Nvidia Blackwell
- NVIDIA Newsroom — NVIDIA Announces Financial Results for Second Quarter Fiscal 2027
- 24/7 Wall St. — OpenAI's Custom Chip Embarrasses Nvidia, While Company Vows to Keep Buying From It
- Deloitte — The dilemma posed by the semiconductor AI supercycle (Global Economic Review, August 2026)