LLM engine from scratch · Part 5/7

Going to the GPU: from 230 to 1,300 tok/s

2026-11-21 · Servloci Engineering

The RTX 3060 has about 360 GB/s of memory bandwidth, five to six times what the CPU gets from DDR5. By Part 1's formula that's a 5–6× higher ceiling for single-stream decoding, and far more arithmetic for batches. This part covers adding a CUDA stage to qpipe: the toolchain, four kernel iterations with numbers for each, and what happened when the CPU and GPU worked as one pipeline.

Getting nvcc without sudo

The machine had the NVIDIA driver (CUDA 13.2) but no toolkit. Rather than touch the system package manager, I used NVIDIA's official pip wheels installed into a private folder. Two lessons from that:

Matching the toolkit (13.2) to the driver avoids any forward-compatibility questions, and compiling straight to SASS with -arch=sm_86 means nothing is JIT-compiled at runtime.

The GPU also had 10 GB of its 12 GB in use by other local services. Stopping two of them freed 9.4 GB, enough for the 1.5B model at Q8 plus KV caches, and later a 4B model and its draft together.

How the GPU fits the pipeline

A GPU stage is a pipeline stage like any other (Part 4). Instead of a team of threads it has one driver thread, pinned to CPU 23 (the SMT sibling of core 11, so it doesn't steal a full core from a CPU stage). It pops a group id, copies the rows' activations to the device, runs its layers, and copies the result back, or, if it holds the lm_head, returns tokens.

A step is about 18 kernel launches per layer: RMSNorm, activation quantize, the three QKV matmuls, bias + q/k norm + RoPE + KV store, attention, the output projection, a residual add, the second RMSNorm, gate and up matmuls, SiLU-multiply, the down matmul, and another residual.

Kernel v1: one warp per row (slower than the CPU)

The first matmul kernel was the obvious one: each warp computes one output row, each lane loads 16 bytes of weights, dequantizes them to floats, and does FMAs against up to 8 activation rows.

Qwen2.5-1.5B Q8 tok/s
1 stream 145
batch 8 239
batch 32 231
prefill, 685-token prompt 292

Single-stream decoding was great immediately: 145 tok/s is about 275 GB/s of weight traffic. But batching barely helped, and prefill was slower than the CPU (292 vs 434 tok/s). Each weight was loaded once per output row, and then every activation value was loaded again for every row: one FMA per float loaded from L1. The kernel was bound by load bandwidth, not math.

Kernel v2: register tiling

Give each warp 4 weight rows instead of 1. Each 16-float activation load now feeds 4 × 16 FMAs instead of 16:

template <int Q4, int RW, int NB>   // RW weight rows per warp, NB activation rows per block
__global__ void __launch_bounds__(256) k_matmul(...) {
    int warp = threadIdx.x >> 5, lane = threadIdx.x & 31;
    int r0 = (blockIdx.y * 8 + warp) * RW;
    float acc[RW][NB] = {};
    for (int u = lane; u < units; u += 32) {      // 16-byte weight units, coalesced
        float w[RW][16];
        for (int r = 0; r < RW; r++) { /* load 16 weights of row r0+r, dequantize */ }
        for (int b = 0; b < NB; b++) {
            float xv[16];  /* 4 × float4 loads */
            for (int r = 0; r < RW; r++)
                for (int j = 0; j < 16; j++) acc[r][b] = fmaf(w[r][j], xv[j], acc[r][b]);
        }
    }
    /* warp_sum each acc, lane 0 writes */
}
v1 v2
batch 8 239 562
batch 32 231 556
prefill 292 680

2.35× at batch 8. But still flat beyond 8.

Kernel v3: order the grid for L2

Each block handles 32 weight rows and 8 activation rows. With 32 activation rows there are 4 block columns, and CUDA launches blocks with blockIdx.x varying fastest. In v2, x was the weight-row block, so the four blocks that share the same weight rows were scheduled far apart in time. By the time the second one ran, its weights had been evicted from the 3 MB L2, and every 8-row slice re-streamed the full model from DRAM.

The fix is one swap: activation chunk on x, weight rows on y. Blocks that need the same weights now run together, and the weights are read from DRAM once.

int r0 = (blockIdx.y * 8 + warp) * RW;     // weight rows: slow axis
int b0 = blockIdx.x * NB;                  // activation chunk: fast axis

Batch 16 went to 640 tok/s and prefill to 831. Then one more small fix: at batch 32, every step copied the full logits back to the host (32 × 151,936 floats = 19 MB over PCIe) so the CPU could find the argmax. A 30-line k_argmax kernel returns just the token ids for greedy rows. Batch 16: 675 tok/s.

Kernel v4: int8 dp4a

The kernel was now compute-bound on the wrong thing: converting every weight to float and doing one FMA per weight per row. Q4 was even slower than Q8, because it had more unpacking to do. The fix mirrors the CPU: quantize the activations to int8 on the GPU and use __dp4a, which multiplies four int8 pairs and accumulates into an int32 in one instruction.

Activation quantization is one warp per 32-wide block: warp-max for the scale, round, and a half-warp sum for the Q4 correction:

__global__ void k_quant(int8_t *xq, float *xd, int *xs16, const float *x, int in) {
    ...
    float v = x[o], amax = warp_max(fabsf(v)), d = amax / 127.f, id = d ? 1.f / d : 0.f;
    int q = __float2int_rn(v * id), s = q;
    xq[o] = (int8_t)q;
    for (int k = 8; k; k >>= 1) s += __shfl_xor_sync(0xffffffff, s, k);   // stays within each half-warp
    if (lane == 0)  { xd[...] = d; xs16[... * 2] = s; }
    if (lane == 16) xs16[... * 2 + 1] = s;
}

For Q8 weights there's a neat trick at upload time. The CPU layout stores w + 128 as an unsigned byte. Flipping the top bit turns that back into the signed byte w, exactly:

for (size_t i = 0; i < qb; i++) t[i] = m.q[i] ^ 0x80;   // (w + 128) ^ 0x80 == w as int8

So the GPU Q8 path needs no correction at all: signed × signed dp4a. Q4 nibbles are w + 8, unpacked with a mask and a shift, and corrected with 8 × Σx per 16 elements:

int4 lo = make_int4(wv.x & m, wv.y & m, wv.z & m, wv.w & m);              // m = 0x0F0F0F0F
int4 hi = make_int4((wv.x >> 4) & m, (wv.y >> 4) & m, (wv.z >> 4) & m, (wv.w >> 4) & m);
acc = fmaf((float)(dot16(lo, xl) - cl), dl * sxl, acc);
acc = fmaf((float)(dot16(hi, xh) - ch), dh * sxh, acc);

Per 16 weights per row, that's 4 dp4a and 2 float ops, instead of 16 conversions and 16 FMAs. Activation loads also shrink 4×, from floats to bytes.

GPU, tok/s fp32 kernel dp4a
Q8: 1 stream / batch 16 / batch 32 144 / 675 / 663 149 / 1,149 / 1,137
Q4: 1 stream / batch 16 / batch 32 171 / 503 / 478 205 / 1,243 / 1,238
Prefill, 685 tokens (Q8 / Q4) 831 1,306 / 1,438

Output was byte-identical to the CPU for Q8, and matched the CPU's Q4 text for Q4.

CUDA graphs

Q4 single-stream decoding was 205 tok/s against a ceiling around 340. About 510 small kernels per token, each with a few microseconds of launch gap, leave the GPU idle between them. CUDA graphs record a whole sequence of launches once and replay it with one call.

The constraint is that a captured graph bakes in every pointer. Host buffers differ per request, so every step first copies its inputs into fixed pinned staging buffers, and the graph reads from and writes to those. Graphs are cached per (rows, logit rows, greedy) shape, and only used for steps of 64 rows or fewer; big prefill steps are long enough that launch overhead doesn't matter.

int key = n | nw << 10 | greedy << 20;
if (miss) {
    cudaStreamBeginCapture(g->s, cudaStreamCaptureModeThreadLocal);
    enqueue(g, n, nw, greedy);              // same code path as the uncaptured version
    cudaStreamEndCapture(g->s, &gr);
    cudaGraphInstantiate(&g->gexec[k], gr, 0);
}
cudaGraphLaunch(g->gexec[k], g->s);

Positions and stream ids live in device memory, so the same graph works for every decoding step of the same shape. The gain was smaller than hoped: Q4 single stream 197 → 216 tok/s, batch 16 1,293 → 1,347. Launch overhead was real but not dominant; Q8 single stream was already at about 280 GB/s.

fp16 KV cache and a fused RoPE kernel

When Qwen3-4B arrived (Part 6), the KV cache became the memory problem: 36 layers × 8 KV heads × 128 dims × 2 (K and V) × 4 bytes is 295 KB per token per stream. Storing K and V as fp16 halves that, with no visible change in output.

Qwen3 also needs a per-head RMSNorm on q and k before RoPE. That became one fused kernel: one warp per head, each lane holding elements lane + 32t. For a 128-wide head, NeoX RoPE pairs element i with i + 64, which is t + 2 in the same lane, so the rotation needs no shuffles at all:

for (int t = 0; t < PER / 2; t++) {           // PER = hd / 32
    float sn, cs, x0 = x[t], x1 = x[t + PER / 2];
    sincosf(p * inv[lane + 32 * t], &sn, &cs);
    x[t] = x0 * cs - x1 * sn;
    x[t + PER / 2] = x0 * sn + x1 * cs;
}

Pairing the CPU with the GPU

The pipeline from Part 4 lets one stage be the CPU and another the GPU: -c 0-11/g runs the first layers on 12 CPU cores and the rest on the GPU.

My first auto-split put the GPU first and gave the CPU the last layer plus the lm_head, which is 15% of the model's weights on the slow device. One stream dropped to 94 tok/s, well below GPU-only. Putting the CPU first and keeping the head on the GPU fixed that.

tok/s CPU only GPU only CPU + GPU
Q8, 1 stream 32 157 98
Q4, 1 stream 50 216 150
Q8, 32 streams about 300 1,147 1,363 (+19%)
Q4, 32 streams about 337 1,275 1,467 (+15%)

The CPU+GPU pipeline wins only with many streams in flight. For one stream, each token passes through both devices in sequence, and the CPU portion is 5× slower, so it just adds latency. With two groups of 16 streams, the CPU works on one group while the GPU works on the other, adding roughly the CPU's share of throughput. The best split gave the CPU layers 0–5 (about a fifth of the work, matching its fifth of the speed).

That's the honest result for this hardware: for one user, run everything on the GPU. For a busy server, let the CPU take the first few layers.

LLM engine from scratch series

  1. The Physics of Token Speed: Why Memory Bandwidth Sets the Limit
  2. Reading GGUF and Running Qwen by Hand
  3. CPU Kernels: Q8, Q4 and AVX-512 VNNI
  4. Threads, Pinning and the Pipeline
  5. Going to the GPU: from 230 to 1,300 tok/s
  6. Speculative Decoding: More Tokens per Weight Read
  7. Serving It: OpenAI API, Tool Calling and KV Reuse
← Part 4Part 6 →