LLM engine from scratch · Part 3/7

CPU Kernels: Q8, Q4 and AVX-512 VNNI

2026-11-21 · Servloci Engineering

Part 1 showed that single-stream decoding is limited by memory bandwidth. So why write fast kernels at all? Because the moment you batch, whether that's several users, a prompt being prefilled, or a speculative verification pass, each weight is reused many times and the arithmetic becomes the bottleneck. This part follows the CPU matrix kernel through three versions, with the numbers for each, plus the 4-bit path and two experiments that didn't help.

All numbers are Qwen2.5-1.5B on the 12 physical cores of a Ryzen 9 7900X, generating 128 tokens.

int8 × int8, not float

The weights are Q8_0: blocks of 32 signed bytes with one fp16 scale. The obvious kernel dequantizes weights to floats and does float FMAs. The faster approach, which llama.cpp also uses, is to quantize the activation vector to int8 too, with its own per-block scale, and do the dot product in integers:

dot(w, x) = Σ_blocks  d_w · d_x · Σ_{i<32} q_w[i] · q_x[i]

The inner sum is pure int8 multiply-accumulate, which is exactly what AVX-512 VNNI's vpdpbusd does: multiply 64 pairs of bytes and add groups of four into 16 int32 lanes, in one instruction.

The catch: vpdpbusd multiplies an unsigned byte by a signed byte. Both our operands are signed.

Version 1: the sign trick

The classic workaround moves the sign from the weight onto the activation: |w| × (x · sign(w)) = w × x.

__m256i aw = _mm256_sign_epi8(wv, wv);   // |w|, now usable as unsigned
__m256i sx = _mm256_sign_epi8(xv, wv);   // x * sign(w)
__m256i s  = _mm256_dpbusd_epi32(_mm256_setzero_si256(), aw, sx);
__m256 d   = _mm256_set1_ps(h2f(w[b].d) * h2f(x[b].d));
acc = _mm256_fmadd_ps(d, _mm256_cvtepi32_ps(s), acc);

One 32-byte block per iteration, two sign instructions, two fp16→fp32 conversions for the scales, then a horizontal sum at the end of each row. The matmul loop is row-outer: each weight row is loaded once and dotted with every activation in the batch while it's hot in L1.

for (int r = r0; r < r1; r++) {
    const blk_q8 *wr = W + (size_t)r * nb;
    for (int b = 0; b < B; b++)
        out[(size_t)b * rows + r] = dot_q8(wr, xq + (size_t)b * nb, nb);
}
Config tok/s
1 stream, 12 cores 30.5
batch 8 138.7

That single-stream number is already about 58 GB/s of weight traffic, close to the RAM limit. Batch 8 shows what's left on the table: only 4.5× faster with 8× the work, and about 430 GOPS of int8 work against a theoretical peak in the TOPS range.

Version 2: repack, correct, tile

Three changes, each removing work from the inner loop.

1. Repack weights so 64 bytes = 2 blocks

In ggml's layout, each block's scale sits between the blocks' bytes, so you can't load 64 contiguous weight bytes. qpipe repacks every weight matrix once at load time into two arrays: the quantized bytes, contiguous, and the fp16 scales, separate. Same total size, but now a single 512-bit load covers two blocks.

2. Store weights as w + 128 and correct on the activation side

Instead of fixing signs per instruction, store each weight as the unsigned byte w + 128. Then:

Σ (w + 128) · x  =  Σ w·x  +  128 · Σ x

The unwanted term depends only on the activation. Each vpdpbusd lane sums 4 products, so for every group of 4 activation bytes we precompute c = −128 × (x₀+x₁+x₂+x₃) once, when the activation is quantized, and use it as the accumulator's starting value:

__m512i dp = _mm512_dpbusd_epi32(cv, wv[i], xv);   // = Σ w·x exactly, no sign instructions

The activation is quantized once per row and reused against thousands of weight rows, so the correction is essentially free. The per-lane float scales are also precomputed and stored expanded (8 lanes per block), so the inner loop never converts fp16.

static inline void act_quant(qact *a, int b, const float *x, int n) {
    ...
    for (int k = 0; k < n / QK; k++) {
        __m512 v0 = _mm512_loadu_ps(v), v1 = _mm512_loadu_ps(v + 16);
        float amax = _mm512_reduce_max_ps(_mm512_max_ps(_mm512_abs_ps(v0), _mm512_abs_ps(v1)));
        float d = h2f(f2h(amax / 127.f)), id = amax ? 127.f / amax : 0.f;
        __m512i i0 = _mm512_cvtps_epi32(_mm512_mul_ps(v0, _mm512_set1_ps(id)));
        ...
        _mm_storeu_si128((__m128i *)(q + k * QK), _mm512_cvtsepi32_epi8(i0));
        _mm256_storeu_ps(s + k * 8, _mm256_set1_ps(d));     // one scale per dpbusd lane
    }
    for (int j = 0; j < n / 4; j++)
        c[j] = -128 * (q[4*j] + q[4*j+1] + q[4*j+2] + q[4*j+3]);
}

3. A register tile: 2 rows × 4 streams

Each weight load is used for 4 activation rows, and each activation load for 2 weight rows. With 512-bit registers that's 8 accumulators plus temporaries, comfortably inside the 32 zmm registers.

for (int p = 0; p < np; p++) {                  // 64 weights per step
    for (int i = 0; i < R; i++) {               // R weight rows
        wv[i] = _mm512_loadu_si512(wq[i] + 64 * p);
        ws[i] = wscale(wd[i] + 2 * p);          // 2 fp16 scales → 16 lanes
    }
    for (int j = 0; j < NB; j++) {              // NB activation rows
        __m512i xv = _mm512_loadu_si512(X->q + b * in + 64 * p);
        __m512i cv = _mm512_loadu_si512(X->c + b * ln + 16 * p);
        __m512  xs = _mm512_loadu_ps(X->s + b * ln + 16 * p);
        for (int i = 0; i < R; i++) {
            __m512i dp = _mm512_dpbusd_epi32(cv, wv[i], xv);
            acc[i][j] = _mm512_fmadd_ps(_mm512_cvtepi32_ps(dp), _mm512_mul_ps(ws[i], xs), acc[i][j]);
        }
    }
}

R and NB are template-like constants: the function is always_inline, and a switch calls it with literal values, so the compiler fully unrolls each combination.

Config v1 v2 Change
1 stream 30.5 29.0 same (bandwidth bound)
batch 8 138.7 201.7 +45%
2 stages × batch 8 143.8 221.9 +54%
batch 16 – 281.5
batch 32 – 309.4
prefill, batch 16 140.7 293.7 +109%

Exactly the pattern Part 1 predicts: big gains where reuse exists, nothing for a single stream.

The 4-bit path

To make single-stream decoding faster, read fewer bytes. qpipe's -4 flag requantizes the layer weights to Q4_0 at load time, using ggml's exact rounding:

float d = mx / -8.f, id = d ? 1.f / d : 0.f;       // mx = the value with the largest |x|
int q = (int)(v[i] * id + 8.5f);                    // stored as 0..15, meaning q - 8
qq[i] = q < 0 ? 0 : q > 15 ? 15 : q;

The nibble layout is chosen for the kernel. Within each 64-weight chunk, byte j holds element j in its low nibble and element j+32 in its high nibble. One 32-byte load, one shift and one mask produce 64 consecutive weights in order:

__m256i v = _mm256_loadu_si256((const __m256i *)(q + 32 * p));
__m512i w = _mm512_inserti64x4(_mm512_castsi256_si512(v), _mm256_srli_epi16(v, 4), 1);
return _mm512_and_si512(w, _mm512_set1_epi8(0x0F));        // values 0..15 = w + 8

The stored value is w + 8, so the correction term is −8 × Σx instead of −128 × Σx. Both are multiples of 16, so the Q4 correction is just the Q8 one shifted right by 4, exactly:

if (Q4) cv = _mm512_srai_epi32(cv, 4);    // -128*Σx  →  -8*Σx

One kernel handles both formats. The lm_head stays 8-bit, as llama.cpp does by default, because output quality is more sensitive there.

Config Q8 Q4 Change
1 stream 30.6 49.9 +63%
batch 8 206 265 +29%
batch 16 276 316 +15%
batch 32 302 337 +12%

The gain shrinks as the batch grows, because large batches are compute-limited and Q4 only saves memory traffic (and costs a little unpacking). Quality stayed good: answers were still correct and coherent.

Experiments that didn't help

A 4×4 tile. If 2 rows × 4 streams is good, 4 rows × 4 streams should save even more activation loads. It didn't: 305 tok/s at batch 32 versus 319 for the 2-row tile. Sixteen accumulators plus four weight registers plus four scale vectors leaves too little room, and the extra pressure cancelled the saved loads. Reverted, kept as a compile-time option (-DTILE_R=4).

Vectorized attention. Attention was still scalar C. Rewriting the q·k dot products and the weighted value sum with AVX-512 improved a 900-token generation by only 2% (27.9 → 28.5 tok/s). With just 2 KV heads, attention is a small slice of the work for this model. It matters more for longer contexts and models with more KV heads, so it stayed.

Where the CPU tops out

At batch 32 the kernel sustains about 0.5 tera-MAC/s. Look at the inner loop: for every 64 multiply-adds there is one integer instruction (vpdpbusd) and three float instructions (convert, scale multiply, FMA). On Zen 4, 512-bit operations are executed as two 256-bit halves, so those float ops are the real cost. The fundamental limit is the per-block scale: every 32 weights you have to leave the integer domain and apply a float scale.

That's a reasonable place to stop on the CPU. The bigger wins come from changing the shape of the work: batching the prompt (Part 4), moving to the GPU (Part 5), and speculative decoding (Part 6).

LLM engine from scratch series

  1. The Physics of Token Speed: Why Memory Bandwidth Sets the Limit
  2. Reading GGUF and Running Qwen by Hand
  3. CPU Kernels: Q8, Q4 and AVX-512 VNNI
  4. Threads, Pinning and the Pipeline
  5. Going to the GPU: from 230 to 1,300 tok/s
  6. Speculative Decoding: More Tokens per Weight Read
  7. Serving It: OpenAI API, Tool Calling and KV Reuse
← Part 2Part 4 →