To write one token, the processor reads the model’s weights from memory once. So, to a first approximation:
tokens per second ≈ memory bandwidth ÷ bytes read per token × efficiencyA 4-bit 8B model is about 5 GB. On a card with 500 GB/s of bandwidth the ceiling is around 100 tokens per second before efficiency losses. A 4-bit 70B model is about 40 GB, so the same card would manage roughly 12 if it could hold it. Real speeds come in below the ceiling because the processor never reaches full bandwidth; our estimates assume about 72% for NVIDIA, 62% for Apple silicon and 62% for AMD.
What this explains
- Why a smaller quantization is faster. Fewer bytes to read per token.
- Why a Mac can beat a PC with more compute. Unified memory on higher-end Macs has high bandwidth, and a lot of it (Macs).
- Why system RAM is slow for models. Typical desktop RAM offers a fraction of a graphics card’s bandwidth, so every offloaded layer drags the speed down (offload).
- Why mixture-of-experts models are fast. They read only the active experts per token (MoE).
- Why two cards do not double the speed. Layers are split across cards, so each token still passes through all of them in turn.
Bandwidth is not the whole story: prefill speed depends on compute (two speeds), and software matters too. But when you compare two machines for chat, bandwidth and memory size are the numbers to look at first. Every hardware page lists them, and you can line machines up on Compare.