The RTX 3060 has about 360 GB/s of memory bandwidth, five to six times what the CPU gets from DDR5. By Part 1's formula that's a 5–6× higher ceiling for single-stream decoding, and far more arithmetic for batches. This part covers adding a CUDA stage to qpipe: the toolchain, four kernel iterations with numbers for each, and what happened when the CPU and GPU worked as one pipeline.
Getting nvcc without sudo
The machine had the NVIDIA driver (CUDA 13.2) but no toolkit. Rather than touch the system package manager, I used NVIDIA's official pip wheels installed into a private folder. Two lessons from that:
- The CUDA 12
nvidia-cuda-nvcc-cu12wheel ships onlyptxas, not thenvccdriver. The CUDA 13 wheels are complete:nvidia-cuda-nvcc,nvidia-cuda-runtime,nvidia-cuda-cccl,nvidia-cuda-crtandnvidia-nvvm(forciccand libdevice), all landing in onenvidia/cu13tree with a workingnvcc.profile. pip install --target DIR --upgradereplaces shared top-level folders. Installingnvidia-nvvmon its own wiped the previously installednvidia/packages. Install all five in a single command.
Matching the toolkit (13.2) to the driver avoids any forward-compatibility questions, and compiling straight to SASS with -arch=sm_86 means nothing is JIT-compiled at runtime.
The GPU also had 10 GB of its 12 GB in use by other local services. Stopping two of them freed 9.4 GB, enough for the 1.5B model at Q8 plus KV caches, and later a 4B model and its draft together.
How the GPU fits the pipeline
A GPU stage is a pipeline stage like any other (Part 4). Instead of a team of threads it has one driver thread, pinned to CPU 23 (the SMT sibling of core 11, so it doesn't steal a full core from a CPU stage). It pops a group id, copies the rows' activations to the device, runs its layers, and copies the result back, or, if it holds the lm_head, returns tokens.
- Weights are uploaded in the same repacked layout the CPU uses (Part 3), so both backends share one loader.
- Host activation buffers are allocated with
cudaHostAlloc(pinned memory), so transfers run at full DMA speed. - The context uses
cudaDeviceScheduleBlockingSync: the driver thread sleeps while the GPU works instead of spinning on a core.
A step is about 18 kernel launches per layer: RMSNorm, activation quantize, the three QKV matmuls, bias + q/k norm + RoPE + KV store, attention, the output projection, a residual add, the second RMSNorm, gate and up matmuls, SiLU-multiply, the down matmul, and another residual.
Kernel v1: one warp per row (slower than the CPU)
The first matmul kernel was the obvious one: each warp computes one output row, each lane loads 16 bytes of weights, dequantizes them to floats, and does FMAs against up to 8 activation rows.
| Qwen2.5-1.5B Q8 | tok/s |
|---|---|
| 1 stream | 145 |
| batch 8 | 239 |
| batch 32 | 231 |
| prefill, 685-token prompt | 292 |
Single-stream decoding was great immediately: 145 tok/s is about 275 GB/s of weight traffic. But batching barely helped, and prefill was slower than the CPU (292 vs 434 tok/s). Each weight was loaded once per output row, and then every activation value was loaded again for every row: one FMA per float loaded from L1. The kernel was bound by load bandwidth, not math.
Kernel v2: register tiling
Give each warp 4 weight rows instead of 1. Each 16-float activation load now feeds 4 × 16 FMAs instead of 16:
template <int Q4, int RW, int NB> // RW weight rows per warp, NB activation rows per block
__global__ void __launch_bounds__(256) k_matmul(...) {
int warp = threadIdx.x >> 5, lane = threadIdx.x & 31;
int r0 = (blockIdx.y * 8 + warp) * RW;
float acc[RW][NB] = {};
for (int u = lane; u < units; u += 32) { // 16-byte weight units, coalesced
float w[RW][16];
for (int r = 0; r < RW; r++) { /* load 16 weights of row r0+r, dequantize */ }
for (int b = 0; b < NB; b++) {
float xv[16]; /* 4 × float4 loads */
for (int r = 0; r < RW; r++)
for (int j = 0; j < 16; j++) acc[r][b] = fmaf(w[r][j], xv[j], acc[r][b]);
}
}
/* warp_sum each acc, lane 0 writes */
}
| v1 | v2 | |
|---|---|---|
| batch 8 | 239 | 562 |
| batch 32 | 231 | 556 |
| prefill | 292 | 680 |
2.35× at batch 8. But still flat beyond 8.
Kernel v3: order the grid for L2
Each block handles 32 weight rows and 8 activation rows. With 32 activation rows there are 4 block columns, and CUDA launches blocks with blockIdx.x varying fastest. In v2, x was the weight-row block, so the four blocks that share the same weight rows were scheduled far apart in time. By the time the second one ran, its weights had been evicted from the 3 MB L2, and every 8-row slice re-streamed the full model from DRAM.
The fix is one swap: activation chunk on x, weight rows on y. Blocks that need the same weights now run together, and the weights are read from DRAM once.
int r0 = (blockIdx.y * 8 + warp) * RW; // weight rows: slow axis
int b0 = blockIdx.x * NB; // activation chunk: fast axis
Batch 16 went to 640 tok/s and prefill to 831. Then one more small fix: at batch 32, every step copied the full logits back to the host (32 × 151,936 floats = 19 MB over PCIe) so the CPU could find the argmax. A 30-line k_argmax kernel returns just the token ids for greedy rows. Batch 16: 675 tok/s.
Kernel v4: int8 dp4a
The kernel was now compute-bound on the wrong thing: converting every weight to float and doing one FMA per weight per row. Q4 was even slower than Q8, because it had more unpacking to do. The fix mirrors the CPU: quantize the activations to int8 on the GPU and use __dp4a, which multiplies four int8 pairs and accumulates into an int32 in one instruction.
Activation quantization is one warp per 32-wide block: warp-max for the scale, round, and a half-warp sum for the Q4 correction:
__global__ void k_quant(int8_t *xq, float *xd, int *xs16, const float *x, int in) {
...
float v = x[o], amax = warp_max(fabsf(v)), d = amax / 127.f, id = d ? 1.f / d : 0.f;
int q = __float2int_rn(v * id), s = q;
xq[o] = (int8_t)q;
for (int k = 8; k; k >>= 1) s += __shfl_xor_sync(0xffffffff, s, k); // stays within each half-warp
if (lane == 0) { xd[...] = d; xs16[... * 2] = s; }
if (lane == 16) xs16[... * 2 + 1] = s;
}
For Q8 weights there's a neat trick at upload time. The CPU layout stores w + 128 as an unsigned byte. Flipping the top bit turns that back into the signed byte w, exactly:
for (size_t i = 0; i < qb; i++) t[i] = m.q[i] ^ 0x80; // (w + 128) ^ 0x80 == w as int8
So the GPU Q8 path needs no correction at all: signed × signed dp4a. Q4 nibbles are w + 8, unpacked with a mask and a shift, and corrected with 8 × Σx per 16 elements:
int4 lo = make_int4(wv.x & m, wv.y & m, wv.z & m, wv.w & m); // m = 0x0F0F0F0F
int4 hi = make_int4((wv.x >> 4) & m, (wv.y >> 4) & m, (wv.z >> 4) & m, (wv.w >> 4) & m);
acc = fmaf((float)(dot16(lo, xl) - cl), dl * sxl, acc);
acc = fmaf((float)(dot16(hi, xh) - ch), dh * sxh, acc);
Per 16 weights per row, that's 4 dp4a and 2 float ops, instead of 16 conversions and 16 FMAs. Activation loads also shrink 4×, from floats to bytes.
| GPU, tok/s | fp32 kernel | dp4a |
|---|---|---|
| Q8: 1 stream / batch 16 / batch 32 | 144 / 675 / 663 | 149 / 1,149 / 1,137 |
| Q4: 1 stream / batch 16 / batch 32 | 171 / 503 / 478 | 205 / 1,243 / 1,238 |
| Prefill, 685 tokens (Q8 / Q4) | 831 | 1,306 / 1,438 |
Output was byte-identical to the CPU for Q8, and matched the CPU's Q4 text for Q4.
CUDA graphs
Q4 single-stream decoding was 205 tok/s against a ceiling around 340. About 510 small kernels per token, each with a few microseconds of launch gap, leave the GPU idle between them. CUDA graphs record a whole sequence of launches once and replay it with one call.
The constraint is that a captured graph bakes in every pointer. Host buffers differ per request, so every step first copies its inputs into fixed pinned staging buffers, and the graph reads from and writes to those. Graphs are cached per (rows, logit rows, greedy) shape, and only used for steps of 64 rows or fewer; big prefill steps are long enough that launch overhead doesn't matter.
int key = n | nw << 10 | greedy << 20;
if (miss) {
cudaStreamBeginCapture(g->s, cudaStreamCaptureModeThreadLocal);
enqueue(g, n, nw, greedy); // same code path as the uncaptured version
cudaStreamEndCapture(g->s, &gr);
cudaGraphInstantiate(&g->gexec[k], gr, 0);
}
cudaGraphLaunch(g->gexec[k], g->s);
Positions and stream ids live in device memory, so the same graph works for every decoding step of the same shape. The gain was smaller than hoped: Q4 single stream 197 → 216 tok/s, batch 16 1,293 → 1,347. Launch overhead was real but not dominant; Q8 single stream was already at about 280 GB/s.
fp16 KV cache and a fused RoPE kernel
When Qwen3-4B arrived (Part 6), the KV cache became the memory problem: 36 layers × 8 KV heads × 128 dims × 2 (K and V) × 4 bytes is 295 KB per token per stream. Storing K and V as fp16 halves that, with no visible change in output.
Qwen3 also needs a per-head RMSNorm on q and k before RoPE. That became one fused kernel: one warp per head, each lane holding elements lane + 32t. For a 128-wide head, NeoX RoPE pairs element i with i + 64, which is t + 2 in the same lane, so the rotation needs no shuffles at all:
for (int t = 0; t < PER / 2; t++) { // PER = hd / 32
float sn, cs, x0 = x[t], x1 = x[t + PER / 2];
sincosf(p * inv[lane + 32 * t], &sn, &cs);
x[t] = x0 * cs - x1 * sn;
x[t + PER / 2] = x0 * sn + x1 * cs;
}
Pairing the CPU with the GPU
The pipeline from Part 4 lets one stage be the CPU and another the GPU: -c 0-11/g runs the first layers on 12 CPU cores and the rest on the GPU.
My first auto-split put the GPU first and gave the CPU the last layer plus the lm_head, which is 15% of the model's weights on the slow device. One stream dropped to 94 tok/s, well below GPU-only. Putting the CPU first and keeping the head on the GPU fixed that.
| tok/s | CPU only | GPU only | CPU + GPU |
|---|---|---|---|
| Q8, 1 stream | 32 | 157 | 98 |
| Q4, 1 stream | 50 | 216 | 150 |
| Q8, 32 streams | about 300 | 1,147 | 1,363 (+19%) |
| Q4, 32 streams | about 337 | 1,275 | 1,467 (+15%) |
The CPU+GPU pipeline wins only with many streams in flight. For one stream, each token passes through both devices in sequence, and the CPU portion is 5× slower, so it just adds latency. With two groups of 16 streams, the CPU works on one group while the GPU works on the other, adding roughly the CPU's share of throughput. The best split gave the CPU layers 0–5 (about a fifth of the work, matching its fifth of the speed).
That's the honest result for this hardware: for one user, run everything on the GPU. For a busy server, let the CPU take the first few layers.