Hardware · Memory

How much memory does a model need?

Pick an LLM and a context length. We add up the weights at every quantization, the KV cache that grows with context, and runtime overhead, then show the smallest GPUs and Macs that hold it.

Llama 3.3 70B Instruct at 8K context

Quantization Weights KV cache + overhead Total Smallest hardware that holds it
Q2_K Low 28.2 GB 3.3 GB 31.5 GB Apple M4 Pro (48 GB), Apple M1 Max (64 GB), Apple M4 Max (64 GB) · 83 in all
Q3_K_M Low 34.6 GB 3.3 GB 37.9 GB Apple M1 Max (64 GB), Apple M4 Max (64 GB), NVIDIA RTX PRO 6000 Blackwell · 76 in all
Q4_K_M Good 43.1 GB 3.3 GB 46.4 GB NVIDIA RTX PRO 6000 Blackwell, AMD Ryzen AI Max+ 395 (128 GB), Apple M4 Max (128 GB) · 49 in all
Q5_K_M High 50.1 GB 3.3 GB 53.4 GB NVIDIA RTX PRO 6000 Blackwell, AMD Ryzen AI Max+ 395 (128 GB), Apple M4 Max (128 GB) · 47 in all
Q6_K High 57.9 GB 3.3 GB 61.2 GB NVIDIA RTX PRO 6000 Blackwell, AMD Ryzen AI Max+ 395 (128 GB), Apple M4 Max (128 GB) · 46 in all
Q8_0 Very high 74.8 GB 3.3 GB 78.1 GB NVIDIA RTX PRO 6000 Blackwell, AMD Ryzen AI Max+ 395 (128 GB), Apple M4 Max (128 GB) · 32 in all
F16 Full 141 GB 3.3 GB 144 GB Apple M3 Ultra (256 GB), Apple M3 Ultra (512 GB) · 9 in all

Hardware is listed when its usable memory is at least 8% above the total. Macs count roughly two-thirds to three-quarters of unified memory as usable. Each link opens the full “can it run” breakdown with speed.

The rule of thumb

1 · WeightsParameters × bytes eachA 70B model at 4 bits per weight is about 70 × 0.6 ≈ 42 GB. Lower quantization shrinks it and costs some quality.
2 · KV cacheGrows with contextEvery token you keep in context stores keys and values for each layer. Doubling context doubles this part, so long documents cost real memory.
3 · OverheadLeave headroomThe runtime, the operating system and other apps need memory too. Aim for a total at least 10% under what your GPU or Mac can give a model.

Estimates come from published hardware specs and our open formulas, not from running the model on your machine. Real speed varies with runtime, drivers and settings. Numbers marked “Measured” come from a linked benchmark.