Models read and write in tokens: short words, pieces of longer words, punctuation. As a rule of thumb 100 tokens is about 75 English words. Speed is quoted in tokens per second.
What the numbers feel like
| Tokens per second | Feels like |
|---|---|
| 1–3 | Watching it type. Fine for a batch job you walk away from. |
| 5–8 | About the pace people read. Usable for chat. |
| 10–20 | Comfortable for chat and for most coding help. |
| 30+ | Faster than you can read. Good for agents that make many calls in a row. |
The labels on this site use the same idea. See how the Runs ratings work.
Two speeds, not one
A request has two phases with different limits:
- Prefill reads your prompt. It processes many tokens at once, so it is limited by raw compute. A long document or a coding agent’s big context makes this the part you wait for. It decides the time to first token.
- Decode writes the reply one token at a time. Each token needs the model’s weights read from memory, so it is limited by memory bandwidth. This is the tokens-per-second figure most people quote (why).
That is why a Mac can have a good tokens-per-second figure yet take a while to start answering a very long prompt, while a graphics card with more compute starts sooner.
If you need it faster
- Choose a smaller model or a lower quantization (quantization).
- Keep the whole model in fast memory (offloading).
- Try a mixture-of-experts model (MoE) or speculative decoding (how it works).
- Shorten the context you really need.
Check any model against your own machine with My hardware.