LLM Quantization Explained
What BF16, FP8, Q4, and GGUF mean, how quantization saves memory, and what to check before choosing a local model.
Quantization makes a model smaller by storing its learned numbers with fewer bits. The tradeoff is approximation: the compressed values are less faithful to the originals, which can affect answers.
For local AI, that means fitting a larger model into available memory, leaving more room for context, or reducing the amount of data moved during generation. It does not guarantee better speed or unchanged quality.
What the labels mean
A model's weights are the numbers it learned during training. Precision describes how those numbers are represented.
| Label | Meaning |
|---|---|
| FP32 | 32-bit floating point; relatively high precision |
| FP16 / BF16 | Two different 16-bit floating-point formats; common source-checkpoint formats |
| FP8 | 8-bit floating point; the exact variant and runtime support matter |
| Q8 / Q6 / Q5 / Q4 / Q3 | Quantization families around those bit widths, often mixing formats |
| BPW | Average bits per weight, useful for comparing actual storage |
| GGUF | A container for weights, tokenizer data, and metadata; not a precision level |
FP16 and BF16 both use two bytes per value. BF16 has a wider numeric range; FP16 has finer precision within its range.
FP8 is not Q8, and FP4 is not Q4. Floating-point formats and block quantization encode values differently, even when their nominal bit counts match.
How much memory does quantization save?
Start with:
weight storage in bytes = parameter count × bits per weight ÷ 8
For a model with 27 billion parameters:
| Representation | Theoretical weight storage |
|---|---|
| FP16 / BF16: 16 bits | 54 GB |
| Exactly 8 bits | 27 GB |
| Exactly 4 bits | 13.5 GB |
| Exactly 3 bits | 10.1 GB |
These are calculations in decimal GB, not download sizes or runtime requirements. Real quants include scales, metadata, and some higher-precision tensors. A “4-bit” file can average more than four bits per weight. Check its actual size.
For mixture-of-experts models, count all the weights being stored, including inactive experts. Active parameters describe selected computation, not the full memory bill.
What gets lost?
Imagine replacing a finely marked ruler with one that has fewer marks. Values between the new marks must be rounded or otherwise approximated.
Quantization does something similar. It maps weights into a smaller set of representable values, usually with scales for individual groups. Better methods protect sensitive parts of the model and allocate precision unevenly.
That is why two “Q4” files can differ in quality. Model architecture, calibration data, and the quantization recipe all matter. Less precision introduces numerical error, but a particular benchmark score does not have to fall at every step.
Matching the source model on a few benchmarks also does not prove a quant is lossless. I care about whether it still handles my code edits, structured outputs, and long prompts reliably.
Reading a GGUF filename
A name such as Q4_K_M describes a quantization preset, not a promise about every tensor:
- Q4: a 4-bit-class quantization.
- K: a particular family of block formats.
- M: the medium preset within that family, with some tensors receiving more precision.
IQ3_* and IQ4_* are other quantization families. Their labels alone do not establish which build is best.
For example, ISTA-DASLab's GSQ/RCO builds optimize quantized values and the allocation of formats across tensors. Unsloth's dynamic builds also distribute precision unevenly. Compare the actual file size, methodology, and task results rather than choosing by the digit alone.
Stored precision is separate from compute precision, too. W4A16 means 4-bit weights and 16-bit activations: the intermediate values created during execution. Accumulation and other operations may use higher precision.
The file is only part of the memory bill
Running a model needs more than its weights:
memory ≈ weights + context state + runtime buffers + temporary values
The KV cache stores attention information from previous tokens. Its memory use depends on the architecture, context length, concurrent conversations, and cache format. Hybrid models may also keep recurrent state.
Quantizing weights does not automatically quantize the cache. A Q4 model can still use an FP16 cache. Supported Q8 or Q4 cache modes can save memory, but they introduce a separate quality tradeoff that needs testing, especially on long-context tasks.
Leave room for the operating system and other applications as well. A model file that barely fits is not necessarily a usable setup.
Smaller does not always mean faster
Lower-bit weights reduce memory traffic, which can help token generation. But the runtime needs efficient kernels for that format and hardware; conversion overhead can erase the benefit.
Measure both prefill, which processes the prompt, and decode, which generates the answer. A configuration can improve one without improving the other. Prefill vs. Decode covers that distinction.
How I choose
I compare a few supported quants at the context length I actually use:
- Fit: leave room beyond the weights for context and working memory.
- Quality: test real coding, writing, or tool-use prompts.
- Speed: measure prompt latency and output generation separately.
- Headroom: check longer conversations and concurrent requests.
I choose the smallest quant that still does my work reliably. The useful result is a model that fits, responds well, and completes the task.