What is model quantization: how a 1.5TB AI model shrinks to 4 bits and fits on one server
Model quantization stores an AI model's numbers in fewer bits to cut memory and compute. Meta's 816-billion-parameter Muse Spark Open is about 1.5TB in 16-bit form; a 4-bit build is roughly a quarter of that and, per Meta, runs on one Nvidia B200 node. The cost is some lost precision
The three lines
- Idea — store each number in 8 or 4 bits instead of 16
- Gain — up to ~4x less memory, faster responses, bigger models per GPU
- Cost — precision loss; block scaling like NVFP4 keeps it small
Key questions
- What is quantization in AI
- **Representing model weights with fewer bits.** | Format | Bytes per value | 816B model weights (calc.) | |---|---|---| | FP32 | 4 | ~3.3TB | | BF16 | 2 | ~1.6TB | | FP8 / INT8 | 1 | ~0.8TB | | 4-bit | 0.5 | ~0.4TB + scales |
- What is NVFP4
- **Nvidia's 4-bit floating-point format for Blackwell GPUs.** | Item | Detail | |---|---| | Value | 4 bits (E2M1) | | Block | 16 values share an FP8 scale | | Tensor | One FP32 scale | | Size | ~3.5x smaller than FP16 (Nvidia) |
- Does quantization reduce accuracy
- **Somewhat, depending on model size and method.** Large models at 8 bits lose little; Nvidia claims about 1% for large models at NVFP4 (vendor figure); small models at crude 4-bit can degrade on math and code.
An AI model is a vast table of numbers. Quantization writes those numbers less precisely and far more compactly. Meta's Muse Spark Open, released October 15, 2026, has 816 billion parameters, roughly 1.5TB at two bytes each. Meta also shipped a 4-bit NVFP4 build it says runs on a single Nvidia B200 node.
Think of writing pi as 3.14 instead of 3.14159265: nearly the same answer for most uses, a fraction of the ink.
1. How much smaller
| Format | Per value | 816B weights (calc.) | Typical use |
|---|---|---|---|
| FP32 | 4 bytes | ~3.3TB | Legacy training |
| BF16 / FP16 | 2 bytes | ~1.6TB | Default training and serving |
| FP8 / INT8 | 1 byte | ~0.8TB | Hopper/Blackwell inference |
| NVFP4 | ~4.5 effective bits | ~0.46TB | Blackwell |
| GGUF 4-bit | ~4-5 bits | ~0.4-0.5TB | PCs and Macs (llama.cpp) |
Running a model also needs a KV cache for context: about 266GB for Muse Spark Open at a full 1M tokens in BF16. That is why Meta quotes 700GB-plus of GPU memory for the 4-bit build.
2. Keeping accuracy at 4 bits
Four bits offer only 16 levels, so modern formats add per-block scales.
| NVFP4 | MXFP4 (industry spec) | |
|---|---|---|
| Value | 4-bit E2M1 | 4-bit E2M1 |
| Block size | 16 | 32 |
| Block scale | FP8 (E4M3) | Power of two (E8M0) |
| Tensor scale | FP32 | None |
| Native hardware | Nvidia Blackwell | Multi-vendor |
Smaller blocks mean finer scales and less error. Nvidia says NVFP4 is about 3.5x smaller than FP16 and 1.8x smaller than FP8, with around 1% loss on large models (vendor data). FP8 runs fast on older Hopper chips; NVFP4 needs Blackwell. The format picks the chip generation.
3. Trade-offs
| Gain | Cost |
|---|---|
| 2-4x memory savings | Fine distinctions blur |
| Bigger models per GPU | Possible drops in math, code, long reasoning |
| Faster responses (less memory traffic) | Needs matching hardware |
| Lower power and cost | Results vary by method |
This is also why memory capacity, and suppliers of high-bandwidth memory such as Korea's SK hynix and Samsung, sit at the center of the AI chip race: however small the model gets, it still has to fit.
4. FAQ
| Question | Answer |
|---|---|
| Can a quantized model be restored? | No; download the original weights |
| Total vs active parameters? | MoE models compute a few experts but store all; quantization shrinks storage |
| Quantization vs distillation? | Distillation trains a new smaller model; quantization re-encodes the same one |