Our first AI customer-service product had to answer real customers in Arabic from a business's own files, on a server that already hosts our other products and has no GPU. The first configuration produced one token per second, which is a sentence every half minute. Changing a single number, the CPU thread count, made it twelve to fourteen times faster. This is what we measured, why the obvious setting was wrong, and the handful of other settings that turned a slow demo into something customers can use.
TL;DR
- On a shared CPU, fewer threads can be much faster. With Ollama and a 3B model,
num_thread=8gave about 1 token/s;num_thread=4gave 12–14 tokens/s on the same machine. - Keep the model resident. A cold load cost 60–120 seconds before the first word; we load on boot and re-check every 4 minutes.
- Every app that shares the Ollama instance must send the same
num_ctx, or the model is reloaded on every switch between them. - The prompt, not the answer, is most of the wait. At ~145 tokens/s of prompt reading, every 100 tokens of context costs about 0.7 s before the first word.
- Bigger models were not the fix: a 7B model wrote at about 4 tokens/s, and a very large mixture-of-experts model at 0.1 tokens/s.
The problem
The product is simple to describe: a business uploads spreadsheets and text files, and an assistant answers its customers from those files only, on a web chat, a QR code or a widget. The answer has to come from the business's knowledge, in natural Arabic, within the time a person will wait in a chat window.
The server is one 8-core machine with 31 GB of RAM. It is not dedicated: at the time of the measurement it also served about 21 websites, and the virtual machine showed around 11% CPU steal, time the hypervisor gave to other tenants. We run Ollama bound to localhost, a small chat model (qwen2.5:3b) and a separate embedding model (bge-m3) for retrieval.
Why the usual fix costs too much
The standard answer to "the model is slow" is a GPU or a hosted API. Both are reasonable, and today our Kauthar AI product uses a hosted model by default with the local one as fallback. But for a first product aimed at small businesses, a GPU server was a fixed monthly cost before the first customer, and a per-token API bill grows with every conversation. We wanted to know how far a CPU we already pay for could go.
The second usual answer is "use a better model". We tried that too. In a test of five shop questions, llama3.1:8b answered more accurately than the 3B model but wrote at 9–11 tokens/s, and qwen2.5:7b at about 4 tokens/s. A very large mixture-of-experts model, squeezed into the same CPU with a 14 GB memory cap and 4 cores, answered all five perfectly at 0.1 tokens/s: five to eight minutes per reply. Quality was not the bottleneck. Speed was.
What we tried
The first configuration let the runtime use all 8 cores. That looks like the right choice: more cores, more parallel matrix work. The measured result was about 1 token per second.
We then pinned num_thread to 4 in every request. Same model, same machine, same day: 12–14 tokens per second. We have not run a controlled sweep of every thread count, so we will not claim 4 is optimal on every server. What we can say is that on this shared machine, 8 was dramatically worse, and 4 has stayed in production since.
Our understanding of why: generating a token is a sequence of steps where all worker threads must finish before the next step starts. If one of the 8 threads is descheduled, because a website request arrived or the hypervisor gave the core to someone else, every other thread waits for it. With all cores claimed, a straggler is almost guaranteed on every step. With 4 threads, there is room on the machine for everything else, and the threads that work rarely wait. Token generation on CPU is also limited by memory bandwidth, so extra threads add contention long before they add speed.
The settings we kept
All of this lives in one small HTTP client, so no code path can forget it:
const NUM_THREAD = 4; // mandatory on this server: 8 threads = ~1 token/s
const NUM_CTX = 4096; // must match every other app on the same Ollama
const KEEP_ALIVE = -1; // never unload an idle model
body.options = { num_thread: NUM_THREAD, num_ctx: NUM_CTX,
temperature, num_predict: maxTokens }; // answers capped at 260 tokens
Same num_ctx everywhere. Two of our products share one Ollama and one copy of the model. Ollama treats a different context size as a different runner, so when one app asked for 4096 and another for a different value, the model was reloaded on every switch. We made the value identical in both and wrote the reason next to it.
Keep it warm. A model loaded from disk took 60–120 seconds to produce its first word (117 seconds for gemma3:4b); a resident qwen2.5:3b answered its first word in about 7–8 seconds. A small job loads the chat and embedding models at boot, then every 4 minutes asks Ollama which models are resident (a free call) and reloads any that were evicted, before a customer pays for it.
Shrink the prompt. With a resident model, the remaining wait is reading the prompt. Our prompts were about 1,050–1,090 tokens, read at about 145 tokens/s: 7–8 seconds before the first word. Writing ran at 20–23 tokens/s, so a 130-token answer took about 6 seconds more. We dropped context lines that never carry an answer (related-question hints that help search but not the reply, and lines duplicated across chunks) and capped the business description at 200 characters. We also learned that Ollama's prompt-prefix reuse only helped when almost the whole prompt was identical; a prompt that shared about half its text with the previous one was read in full. The gain came from shorter prompts and caching, not from reordering.
Stream. The web chat streams tokens as they arrive. On WhatsApp, where a message cannot grow, long answers go out in up to three natural chunks while the model is still writing: the first chunk arrived at 12.1 s instead of the full reply at 16.8 s.
Filter the output. The small model occasionally leaked Chinese characters into Arabic replies. A streaming filter strips CJK ranges before anything reaches the customer.
Skip the model when you can. Fixed answers, buttons and small talk never touch the model at all. Only questions that need retrieval and wording reach it.
Measured result
With these settings the production server handles about 10 model answers per minute with two requests in parallel. One request at a time takes 5–7 seconds; four at once, 19–24 seconds each; twelve at once, 45–73 seconds. That curve is why the system records how many calls are waiting and shows it on an admin load screen: we know when the CPU has stopped being enough, instead of guessing.
| Setting | Measured |
|---|---|
qwen2.5:3b, 8 threads | ~1 token/s |
qwen2.5:3b, 4 threads | 12–14 tokens/s (later 20–23 on a quieter day) |
qwen2.5:7b | ~4 tokens/s |
llama3.1:8b | 9–11 tokens/s |
| Large MoE model, 4 cores | 0.1 tokens/s |
| Cold model load | 60–120 s before the first word |
Checklist for a CPU-only model server
- Measure tokens per second at several
num_threadvalues on the real, loaded server. Do not assume all cores is best. - Leave CPU for everything else on the machine, and give paying products priority over internal tools.
- Use the same
num_ctxand model options in every app that shares the runtime. - Keep models resident and re-check that they still are.
- Measure prompt length in tokens; it is usually most of the wait.
- Stream, and answer without the model whenever a fixed answer exists.
- Record queue depth, so you know when it is time for a GPU or an API.
Need this?
Kauthar AI is the customer-service assistant we built on these lessons: it answers your customers on WhatsApp, your website and social channels from your own files. If you want a private model on your own server instead, we can measure your hardware and set it up. Talk to GeoTech.