On-device AI explained — what changed to let big models run without a server

For several years, using AI meant borrowing someone else's computer. Your question crossed the internet to a datacenter GPU and the answer came back. Lose connectivity and it stopped. Use it heavily and it billed. Whatever you typed left your hands.

On-device AI removes that round trip. What changed in 2026 is scale: models of 30 billion parameters, not 3 billion, now run on a single laptop — which turns the approach from a demo into an option.

This reference covers what qualifies as on-device, what made it possible, and what still belongs to the server.

1. Two architectures

The last two rows carry the argument. On-device starts responding sooner because nothing travels — but capability and security responsibility are both confined to the hardware in front of you.

2. The three techniques

① Quantization — solves memory. Model parameters are normally stored at 16 or 32 bits each. Thirty billion of them exceeds 55GB at full precision, which does not fit on personal hardware. Storing them at roughly 4 bits cuts the requirement by three-quarters or more. Loosely, it is aggressive rounding, and precision pays for it. Modern schemes reduce the damage by keeping sensitive layers at higher precision and compressing the rest harder.

② Distillation — solves capability. A large model's outputs are used as the teacher for a smaller model. A 30B distilled from something much larger frequently outperforms a 30B trained from scratch. Most of the on-device models released in 2026 are distillations of a bigger sibling.

③ Speculative decoding — solves speed. Fitting in memory is not the same as being usable. Text generation confirms one token at a time, and if that is slow, nothing built on it works. Speculative decoding predicts several tokens at once and rolls back only the incorrect ones — faster when the guesses land, no accuracy penalty when they don't.

The third has become critical for a specific reason: what people want to run locally now is not a chatbot that answers once but an agent, which calls a tool, reads the result and decides again, dozens of times per task. One slow response makes the whole loop unusable.

3. What stays on the server

Three ceilings remain.

Total memory. Quantization shrinks the requirement; it does not remove the limit. Bigger models still mean a bigger machine. Precision. Quantization is a trade. Fine numerical judgment and long chains of reasoning are where the loss becomes visible. Context length. Holding a long document requires memory beyond the parameters themselves, and that portion compresses poorly.

So the shape that is actually emerging is not either/or but division of labor: routine, repetitive and sensitive work handled locally, with harder judgments or long contexts escalated to a server model. Teaching the local model to decide when to escalate is the open problem.

4. Security: better and worse

Better, unambiguously, on data exposure. Input never leaves the device, so interception, vendor log retention and reuse in training all disappear as categories of risk. That is the practical value that matters most — it makes AI usable on material whose export is itself the problem: internal records, contracts, medical files.

Worse on a less-discussed axis. Prompt-injection defense moves onto the device. Prompt injection hides instructions inside a document or web page so the model acts on the attacker's intent rather than the user's, and the risk grows with every file-read or tool-call permission an agent holds.

Published benchmarks put attack success rates for recent models in the 20–40% range. The implication is plain: unsolved. What a vendor previously absorbed with filters and monitoring now has to be handled in a personal computing environment.

5. Common questions

Will it run on my laptop? It depends on model size and memory. A recent Mac with generous unified memory, or a PC with a high-end GPU, brings a 4-bit 30B model into range. Below that, 7B–14B is the realistic target.

Does an NPU help? NPUs are efficient per watt, which matters on battery. But large-model inference today is still driven mainly by GPU throughput and memory bandwidth.

Are open weights and on-device the same thing? No. Open weights means the parameters were published; on-device means the model runs locally. They travel together in practice because running locally requires obtaining the weights. Grades of openness are covered in "What is an open-source AI model".

6. What remains unverified

Quality loss from quantization varies widely by model, task and technique, and no general figure exists. The memory numbers cited follow vendor announcements, and models differ in whether published benchmarks were measured before or after compression. The 20–40% injection range comes from a specific benchmark and does not map directly onto real-world risk.

A shipping example is in today's companion piece "Meta put a 30B agent model on a laptop"; national-scale AI infrastructure is covered in "What is sovereign AI".