LLM engine from scratch · Part 1/7

The Physics of Token Speed: Why Memory Bandwidth Sets the Limit

2026-11-21 · Servloci Engineering

I wanted to answer a simple question: how fast can a normal desktop run a language model if every core, every cache and the GPU are used deliberately instead of by accident? The answer turned into qpipe, an inference engine written from scratch in C and CUDA with no dependencies. It now runs Qwen2.5 and Qwen3 models, has a CPU+GPU pipeline, speculative decoding, an OpenAI-compatible server with tool calling, and KV cache reuse across turns.

This series walks through every layer of it, with the real code and the real numbers, including the experiments that failed. This first part is about the one idea that explains almost everything that follows: generating a token is a memory problem, not a math problem.

The machine

Everything in this series was measured on one desktop:

Part Spec What matters for inference
CPU AMD Ryzen 9 7900X, 12 cores / 24 threads 2 CCDs (chiplets) of 6 cores, 32 MB L3 each, AVX-512 with VNNI
RAM 64 GB dual-channel DDR5 about 60–70 GB/s in practice
GPU NVIDIA RTX 3060, 12 GB about 360 GB/s memory bandwidth, int8 dp4a

Nothing exotic. That is the point: the techniques here are what make cheap hardware useful.

What one token costs

A transformer decoder produces one token at a time. For each new token it runs the full stack of layers once. Each layer multiplies the current activation vector by a set of weight matrices: query, key, value and output projections for attention, then three matrices for the feed-forward block (gate, up, down). At the end, the final hidden state is multiplied by the vocabulary matrix (the lm_head) to get a score for every possible next token.

The important observation: every weight in the model is read once per token. For a 1.5 billion parameter model stored in 8-bit (Q8_0, about 1.06 bytes per weight once you include scales), that is roughly 1.9 GB of memory traffic for every single token.

The arithmetic per token is about 2 operations per weight (one multiply, one add), so about 3 GFLOP for a 1.5B model. A modern CPU core can do tens of GFLOP per second with SIMD. Twelve cores can do hundreds. The math finishes long before the data arrives.

The ceiling formula

That gives a simple upper bound for single-stream decoding:

tokens per second  ≤  memory bandwidth  /  bytes of weights read per token

Plug in the machine:

Model Weights read per token CPU ceiling (about 65 GB/s) GPU ceiling (about 360 GB/s)
Qwen2.5-1.5B, Q8 about 1.9 GB about 34 tok/s about 190 tok/s
Qwen2.5-1.5B, Q4 about 1.0 GB about 65 tok/s about 360 tok/s
Qwen3-4B, Q8 about 4.3 GB about 15 tok/s about 84 tok/s

And here is what qpipe actually measured by the end of the project:

Model CPU, 12 cores GPU Effective bandwidth
Qwen2.5-1.5B, Q8 30.5 tok/s 157 tok/s about 58 GB/s CPU, about 300 GB/s GPU
Qwen2.5-1.5B, Q4 50 tok/s 216 tok/s (lm_head kept at 8-bit)
Qwen3-4B, Q8 12.8 tok/s 62.6 tok/s about 270 GB/s GPU

The measured numbers sit at 75–90% of the ceiling. That is what "done" looks like for single-stream decoding. Past that point, the only ways to go faster are to read fewer bytes per token or to get more than one token out of each read.

The three levers

Every speed-up in this series is one of three things:

  1. Read fewer bytes. Quantization. Going from 8-bit to 4-bit weights nearly doubles the ceiling. This is why the Q4 path took single-stream speed from 30.5 to 50 tok/s on the CPU.
  2. Share each read across many tokens. Batching. If 16 users are generating at once, each weight is loaded once and used 16 times. Now the arithmetic finally matters, and the work shifts from memory-bound to compute-bound. This is where SIMD kernels, tiling, and dp4a earn their keep. On the GPU, batching took the 1.5B model from about 150 tok/s for one user to over 1,300 tok/s total for sixteen.
  3. Guess ahead and verify in bulk. Speculative decoding. A small model guesses several tokens; the big model checks all of them in one pass, which costs about the same as generating one token because the weights are read once. If the guesses are good, you get several tokens per read. This took Qwen3-4B from 66 to 98 tok/s on code.

A fourth lever applies to the prompt, not the output: don't recompute what you already computed. Agent loops resend the same system prompt and tool definitions every turn. Reusing the KV cache across requests cut the time to first token from 478 ms to 56 ms in the final server.

Things that do not help (and why)

Knowing the ceiling also tells you what is pointless:

Prefill is different

There is one phase where compute does matter even for a single user: prefill, processing the prompt. All prompt tokens are known up front, so they can be pushed through each layer as a batch, and every weight load is shared across hundreds of tokens. The first version of qpipe processed prompts one token at a time, at about 30 tok/s. Batching the prompt raised that to 434 tok/s on the CPU and over 1,300 tok/s on the GPU. Same hardware, same model, just respecting the physics.

The plan for qpipe

The engine ended up with this shape:

                         ┌───────────── one "group" of streams ─────────────┐
 HTTP / OpenAI API ──►  stage 0: layers 0..k      ──SPSC queue──►  stage 1: layers k..N + lm_head
 (slots, KV reuse)       CPU team pinned to CCD 0                  CPU team on CCD 1, or the GPU
         ▲                                                                   │
         └────────────────────── sampled tokens feed back ───────────────────┘

The series

  1. The physics of token speed (this part)
  2. Reading GGUF and running Qwen by hand: the file format, the tokenizer, and the forward pass, including what changed from Qwen2 to Qwen3
  3. CPU kernels: Q8 and Q4 formats, AVX-512 VNNI, the repacking trick, and a tiling experiment that failed
  4. Threads, pinning and the pipeline: SPMD stage teams, a spin barrier, a lock-free queue, batched prefill, and why the pipeline gives no single-stream gain
  5. Going to the GPU: from a naive CUDA kernel to int8 dp4a, CUDA graphs, fp16 KV, and pairing CPU with GPU
  6. Speculative decoding: why it is exact, how rollback costs nothing, and adaptive draft length
  7. Serving it: an OpenAI-compatible server, tool calling for Qwen, and KV cache reuse across turns

LLM engine from scratch series

  1. The Physics of Token Speed: Why Memory Bandwidth Sets the Limit
  2. Reading GGUF and Running Qwen by Hand
  3. CPU Kernels: Q8, Q4 and AVX-512 VNNI
  4. Threads, Pinning and the Pipeline
  5. Going to the GPU: from 230 to 1,300 tok/s
  6. Speculative Decoding: More Tokens per Weight Read
  7. Serving It: OpenAI API, Tool Calling and KV Reuse
Part 2 →