I wanted to answer a simple question: how fast can a normal desktop run a language model if every core, every cache and the GPU are used deliberately instead of by accident? The answer turned into qpipe, an inference engine written from scratch in C and CUDA with no dependencies. It now runs Qwen2.5 and Qwen3 models, has a CPU+GPU pipeline, speculative decoding, an OpenAI-compatible server with tool calling, and KV cache reuse across turns.
This series walks through every layer of it, with the real code and the real numbers, including the experiments that failed. This first part is about the one idea that explains almost everything that follows: generating a token is a memory problem, not a math problem.
The machine
Everything in this series was measured on one desktop:
| Part | Spec | What matters for inference |
|---|---|---|
| CPU | AMD Ryzen 9 7900X, 12 cores / 24 threads | 2 CCDs (chiplets) of 6 cores, 32 MB L3 each, AVX-512 with VNNI |
| RAM | 64 GB dual-channel DDR5 | about 60–70 GB/s in practice |
| GPU | NVIDIA RTX 3060, 12 GB | about 360 GB/s memory bandwidth, int8 dp4a |
Nothing exotic. That is the point: the techniques here are what make cheap hardware useful.
What one token costs
A transformer decoder produces one token at a time. For each new token it runs the full stack of layers once. Each layer multiplies the current activation vector by a set of weight matrices: query, key, value and output projections for attention, then three matrices for the feed-forward block (gate, up, down). At the end, the final hidden state is multiplied by the vocabulary matrix (the lm_head) to get a score for every possible next token.
The important observation: every weight in the model is read once per token. For a 1.5 billion parameter model stored in 8-bit (Q8_0, about 1.06 bytes per weight once you include scales), that is roughly 1.9 GB of memory traffic for every single token.
The arithmetic per token is about 2 operations per weight (one multiply, one add), so about 3 GFLOP for a 1.5B model. A modern CPU core can do tens of GFLOP per second with SIMD. Twelve cores can do hundreds. The math finishes long before the data arrives.
The ceiling formula
That gives a simple upper bound for single-stream decoding:
tokens per second ≤ memory bandwidth / bytes of weights read per token
Plug in the machine:
| Model | Weights read per token | CPU ceiling (about 65 GB/s) | GPU ceiling (about 360 GB/s) |
|---|---|---|---|
| Qwen2.5-1.5B, Q8 | about 1.9 GB | about 34 tok/s | about 190 tok/s |
| Qwen2.5-1.5B, Q4 | about 1.0 GB | about 65 tok/s | about 360 tok/s |
| Qwen3-4B, Q8 | about 4.3 GB | about 15 tok/s | about 84 tok/s |
And here is what qpipe actually measured by the end of the project:
| Model | CPU, 12 cores | GPU | Effective bandwidth |
|---|---|---|---|
| Qwen2.5-1.5B, Q8 | 30.5 tok/s | 157 tok/s | about 58 GB/s CPU, about 300 GB/s GPU |
| Qwen2.5-1.5B, Q4 | 50 tok/s | 216 tok/s | (lm_head kept at 8-bit) |
| Qwen3-4B, Q8 | 12.8 tok/s | 62.6 tok/s | about 270 GB/s GPU |
The measured numbers sit at 75–90% of the ceiling. That is what "done" looks like for single-stream decoding. Past that point, the only ways to go faster are to read fewer bytes per token or to get more than one token out of each read.
The three levers
Every speed-up in this series is one of three things:
- Read fewer bytes. Quantization. Going from 8-bit to 4-bit weights nearly doubles the ceiling. This is why the Q4 path took single-stream speed from 30.5 to 50 tok/s on the CPU.
- Share each read across many tokens. Batching. If 16 users are generating at once, each weight is loaded once and used 16 times. Now the arithmetic finally matters, and the work shifts from memory-bound to compute-bound. This is where SIMD kernels, tiling, and
dp4aearn their keep. On the GPU, batching took the 1.5B model from about 150 tok/s for one user to over 1,300 tok/s total for sixteen. - Guess ahead and verify in bulk. Speculative decoding. A small model guesses several tokens; the big model checks all of them in one pass, which costs about the same as generating one token because the weights are read once. If the guesses are good, you get several tokens per read. This took Qwen3-4B from 66 to 98 tok/s on code.
A fourth lever applies to the prompt, not the output: don't recompute what you already computed. Agent loops resend the same system prompt and tool definitions every turn. Reusing the KV cache across requests cut the time to first token from 478 ms to 56 ms in the final server.
Things that do not help (and why)
Knowing the ceiling also tells you what is pointless:
- More threads for a single stream. Once memory bandwidth is saturated, extra cores just wait in line. Twelve cores already pull about 58 GB/s; 24 hyperthreads do not pull more.
- Splitting one stream across CPU and GPU. I built a pipeline that runs the first layers on the CPU and the rest on the GPU. For a single conversation it made things slower (98 tok/s instead of 157), because each token must still pass through both devices in sequence, and the CPU portion runs at a sixth of the GPU's speed. Part 5 shows where it does help: many streams at once.
- Faster kernels for single-stream decode. In Part 3, a carefully tiled AVX-512 kernel improved batched throughput by 45% and single-stream speed by exactly zero. The single stream was never waiting on math.
Prefill is different
There is one phase where compute does matter even for a single user: prefill, processing the prompt. All prompt tokens are known up front, so they can be pushed through each layer as a batch, and every weight load is shared across hundreds of tokens. The first version of qpipe processed prompts one token at a time, at about 30 tok/s. Batching the prompt raised that to 434 tok/s on the CPU and over 1,300 tok/s on the GPU. Same hardware, same model, just respecting the physics.
The plan for qpipe
The engine ended up with this shape:
┌───────────── one "group" of streams ─────────────┐
HTTP / OpenAI API ──► stage 0: layers 0..k ──SPSC queue──► stage 1: layers k..N + lm_head
(slots, KV reuse) CPU team pinned to CCD 0 CPU team on CCD 1, or the GPU
▲ │
└────────────────────── sampled tokens feed back ───────────────────┘
- Stages own a contiguous range of layers. A stage is either a team of threads pinned to specific cores, or a GPU driven by one pinned thread.
- Groups of streams circulate through the stages. While stage 1 works on group A, stage 0 works on group B.
- Rows are the unit of work: a row is one (stream, position, token). Decoding is one row per stream; prefill is many rows per stream. This one abstraction later made batched prefill, speculative verification and KV reuse almost free to add.
The series
- The physics of token speed (this part)
- Reading GGUF and running Qwen by hand: the file format, the tokenizer, and the forward pass, including what changed from Qwen2 to Qwen3
- CPU kernels: Q8 and Q4 formats, AVX-512 VNNI, the repacking trick, and a tiling experiment that failed
- Threads, pinning and the pipeline: SPMD stage teams, a spin barrier, a lock-free queue, batched prefill, and why the pipeline gives no single-stream gain
- Going to the GPU: from a naive CUDA kernel to int8
dp4a, CUDA graphs, fp16 KV, and pairing CPU with GPU - Speculative decoding: why it is exact, how rollback costs nothing, and adaptive draft length
- Serving it: an OpenAI-compatible server, tool calling for Qwen, and KV cache reuse across turns