1. Memory size decides what you can run
A model that fits in video memory runs at full speed; one that does not is slow or does not run at all (why). So buy memory first. Weights alone at Q4_K_M, the usual default:
| Model size | Weights at Q4_K_M |
|---|---|
| 8B | 4.9 GB |
| 14B | 8.5 GB |
| 32B | 19.5 GB |
| 70B | 42.7 GB |
Add the KV cache and about a gigabyte of overhead on top (context). Check an exact model with the memory calculator, or browse what each card can run under Hardware.
2. Bandwidth decides how fast it is
Between two cards with the same memory, the one with more bandwidth generates faster (why). Compare candidates side by side on Compare.
3. Then everything else
- Software support. NVIDIA cards work with the widest range of apps. AMD and Intel cards work in the main llama.cpp-based apps, but check that your app and operating system combination supports the card before buying. The app pages list what each one runs on.
- Two cards. llama.cpp-based apps can split a model across cards, which pools their memory. Speed does not add up: tokens still pass through the layers in turn, so our estimates apply a scaling factor rather than doubling. Two 16 GB cards will hold a bigger model than one, not run a small one twice as fast.
- Laptop cards. A laptop GPU often has less memory and bandwidth than the desktop card with a similar name. Check the memory figure, not the name.
- Used cards. Memory per dollar is often best here. Check the seller’s return policy and remember older cards use more power for the same speed.
- Power and size. Large cards need a strong power supply, extra connectors and case room. Budget for them.
- Not only graphics cards. Macs and some mini PCs share a large memory pool with the processor (memory types, Macs).
For prices and where to buy, see Products. If you already own a machine, run My hardware first, because the best upgrade might be a different quantization.