Hardware · Memory
How much memory does a model need?
Pick an LLM and a context length. We add up the weights at every quantization, the KV cache that grows with context, and runtime overhead, then show the smallest systems and devices that hold it.
MiMo V2.6 Distill Qwen 9B at 8K context
| Quantization | Weights | KV cache + overhead | Total | Smallest hardware that holds it |
|---|---|---|---|---|
Q2_K |
3.8 GB | 0.9 GB | 4.6 GB | |
Q3_K_M |
4.6 GB | 0.9 GB | 5.5 GB | |
Q4_K_M |
5.7 GB | 0.9 GB | 6.6 GB | |
Q5_K_M |
6.7 GB | 0.9 GB | 7.5 GB | |
Q6_K |
7.7 GB | 0.9 GB | 8.6 GB | |
Q8_0 |
10.0 GB | 0.9 GB | 10.8 GB | |
F16 |
18.8 GB | 0.9 GB | 19.7 GB |
The rule of thumb
1 · WeightsParameters × bytes eachA 70B model at 4 bits per weight is about 70 × 0.6 ≈ 42 GB. Lower quantization shrinks it and costs some quality.
2 · KV cacheGrows with contextEvery token you keep in context stores keys and values for each layer. Doubling context doubles this part, so long documents cost real memory.
3 · OverheadLeave headroomThe runtime, the operating system and other apps need memory too. Aim for a total at least 10% under the memory your GPU or device can give a model.
Estimates come from published hardware specs and our open formulas, not from running the model on your machine. Real speed varies with runtime, drivers and settings. Numbers marked “Measured” come from a linked benchmark.