You do not need a graphics card to run a local model, only patience and realistic expectations. Without one, the model lives in system RAM and the CPU does the work. System RAM is far slower than video memory, so speed is set by how much of it the processor must read per token (bandwidth).
A rough estimate
We assume about 60 GB/s for ordinary desktop memory and 80% efficiency. That gives about 10 tokens per second for a 4-bit 8B model (about 5 GB), around 24 for a 3B model (about 2 GB), and well under 2 for a 70B model. Your memory speed and channel count change this: faster, dual-channel memory helps, a single stick hurts.
Getting the most from it
- Choose small, modern models. A recent 3B to 8B model is far better than older models of the same size. The Model Finder on the home page sorts by what runs well on your machine.
- Prefer mixture-of-experts models. They read only a few experts per token, so they run faster on slow memory than dense models of the same total size (MoE).
- Quantize. Q4_K_M is the best balance; going lower hurts quality quickly (quantization).
- Keep the context short. Reading a long prompt on a CPU is slow; the wait before the first word is the biggest annoyance (why).
- Use a CPU-friendly app. llama.cpp, Ollama and LM Studio all run on CPU. Leave a few GB of RAM free for the operating system.
Small decision models (classifiers, rerankers and similar) often run well even on a modest CPU. And if a graphics card is in your plans, start here.