The full quantization ladder
Quantization stores a model's weights at lower numeric precision to shrink file size and memory use. "K-quants" (the _K_ formats below) are the current standard: they mix bit depths internally rather than using one uniform width, which is why a "4-bit" format averages closer to 4.5 bits per weight in practice.
| Format | Bits/weight | 7B size | Quality |
|---|---|---|---|
| FP16 | 16.0 | ~13 GB | Full precision, the inference baseline |
| Q8_0 | ~8.0 | ~6.7 GB | Near-lossless |
| Q6_K | ~6.0 | ~5.15 GB | Almost lossless |
| Q5_K_M | ~5.1 | ~4.45 GB | High quality, the safe balanced choice above Q4 |
| Q4_K_M | ~4.5 | ~3.80 GB | Most popular default; good quality, meaningfully smaller |
| Q3_K_M | ~3.3 | ~3.06 GB | Usable but noticeably below FP16 |
| Q2_K | ~2.6 | ~2.9 GB | Smallest, quality degrades significantly; last resort |
Sizes scale roughly linearly with parameter count: a 32B model at a given format is about 4.6x the size shown above for 7B. Run the exact figure for your model through the hardware calculator rather than scaling by hand.
How much quality actually changes
Quantization loss is usually measured by perplexity, a score for how well a model predicts held-out text; a small increase in perplexity means the model got slightly worse at predicting text it wasn't trained on, which correlates with a general quality drop. The increase from FP16 stays close to zero through Q6_K and Q5_K_M, both considered near-lossless in practice. Q4_K_M is where the tradeoff becomes a real decision rather than a formality: still good quality, but a measurably bigger step down than anything above it. Below Q4, quality loss becomes progressively easier to notice in actual use, not just in a benchmark score.
Choosing the right level for your hardware
Work backward from your available memory rather than picking a format first: take your GPU's total VRAM, subtract roughly 500MB for system overhead, then subtract the memory your target context length needs (see the calculator's context setting for that figure). Whatever's left is your weight budget. Pick the highest quality format from the ladder above that fits inside it.
In practice, most people should treat Q5_K_M as the default when it fits, and drop to Q4_K_M when it doesn't; below Q4_K_M is worth it only when it's genuinely the difference between running a model and not running it at all, not as a routine choice. See every level's size side by side in the quantization calculator instead of doing this math by hand.
Legacy formats: Q4_0, Q5_0, and why to skip them
Older quantization formats without the _K_ naming (Q4_0, Q4_1, Q5_0, Q5_1) still show up in some model repositories. They use a simpler, uniform-precision approach that the K-quant formats have generally superseded on quality per gigabyte. Unless you're using a tool with a specific compatibility requirement for the legacy naming, pick the equivalent K-quant format instead.