Which memory a model actually lives in on a GeForce, a Mac or a mini PC, and what happens when it doesn’t fit.
A model has to sit in memory the processor can read very fast. Which memory that is depends on the machine.
Graphics cardVRAMGeForce, Radeon and Arc cards have their own memory. Fast, but fixed: a 16 GB card holds 16 GB. Two or more cards can pool theirs.
Mac, Ryzen AI, JetsonUnified memoryOne pool shared by CPU and GPU. A 64 GB Mac has far more room for models than a 16 GB graphics card, though the system reserves some, so we plan around about two-thirds to three-quarters of it.
Any PCSystem RAMPlentiful and cheap, but several times slower than VRAM. It is the overflow area when a model is too big for the card.
What happens when a model doesn’t fit
Apps based on llama.cpp (LM Studio, Ollama, Jan and others) can offload: keep some layers on the GPU and run the rest from system RAM. It works, but every token waits on the slower memory, so speed drops sharply. That is what the Slow and Very slow labels on this site mean. vLLM, SGLang and MLX need the whole model in GPU or unified memory.
Speed follows bandwidth
To write one token the processor reads the model’s weights once, so tokens per second is roughly memory bandwidth divided by model size. That is why a Mac with fast unified memory can beat a PC with more raw compute, and why mixture-of-experts models, which read only a few experts per token, run faster than their size suggests. Compare machines side by side on Compare, or see what yours can run on My hardware.
Estimates come from published hardware specs and our open formulas, not from running the model on your machine. Real speed varies with runtime, drivers and settings. Numbers marked “Measured” come from a linked benchmark.
