A model’s weights are billions of numbers. Quantization stores each one in fewer bits, so the file is smaller and each token needs less memory traffic. The cost is a small loss in quality that grows as you go lower.
| Name | Bits | 8B model | 70B model | Quality |
|---|---|---|---|---|
Q2_K | 2 | 3.2 GB | 28 GB | Low |
Q3_K_M | 3 | 3.9 GB | 34 GB | Low |
Q4_K_M | 4 | 4.9 GB | 43 GB | Good |
Q5_K_M | 5 | 5.7 GB | 50 GB | High |
Q6_K | 6 | 6.6 GB | 57 GB | High |
Q8_0 | 8 | 8.5 GB | 74 GB | Very high |
MXFP4 | 4 | 4.6 GB | 40 GB | Native |
F16 | 16 | 16.0 GB | 140 GB | Full |
Reading the names
Q4,Q5,Q8: roughly that many bits per weight.K: the “K-quant” scheme, which spends extra bits where they matter most._M/_S/_L: medium, small or large mix within that scheme.Q4_K_Mis the usual default.MXFP4: a 4-bit floating-point format some models ship in natively, so it costs no extra quality.
Which one should I download?
Take the highest quantization that still fits with headroom and runs at a speed you like. In practice that is Q4_K_M for tight memory, Q5_K_M or Q6_K with room to spare, and Q8_0 only when memory is plentiful. Below 4 bits, quality drops quickly, so prefer a smaller model at 4 bits over a bigger one at 2 bits. Each model page on this site shows the memory for every quantization, and the memory calculator adds your context length.