The original goal of qpipe was a "synced pipeline": split the model's layers into stages, give each stage its own cores, and have each layer group write straight into the next group's input. This part covers how that works, why it does not speed up a single conversation, what it does do, and the change that mattered far more than expected: batched prefill.
The topology
lscpu -e on the Ryzen 9 7900X:
CPU CORE L1d:L1i:L2:L3
0 0 0:0:0:0 ← CCD 0, L3 #0
...
5 5 5:5:5:0
6 6 8:8:8:1 ← CCD 1, L3 #1
...
11 11 13:13:13:1
12 0 0:0:0:0 ← SMT sibling of core 0
...
23 11 13:13:13:1 ← SMT sibling of core 11
Two chiplets (CCDs) of six cores, each with its own 32 MB L3, and one NUMA node. Moving data between CCDs goes over the Infinity Fabric, which is slower than staying inside one L3. So the natural split is one pipeline stage per CCD: -c 0-5/6-11.
Two decisions follow from Part 1:
- Use physical cores, not hyperthreads. The int8 kernels saturate the vector units, and two hyperthreads share one core's units. Twelve threads beat 24.
- Pin every thread.
pthread_setaffinity_npto one CPU each. Unpinned threads migrate, lose their L1/L2 contents, and can land two to a core.
A stage is an SPMD team
Each stage is a team of T pinned threads that all execute the same code (SPMD, single program multiple data). Every matrix multiply is split by output rows; small per-row work (norms, RoPE, residual adds) is split by batch row. Between steps that depend on each other, the team meets at a barrier.
static void split(int n, int tid, int T, int *a, int *b) {
int c = (n + T - 1) / T;
c = (c + 7) & ~7; // multiples of 8 rows, friendly to the 2-row tile
*a = tid * c; *b = *a + c;
if (*a > n) *a = n;
if (*b > n) *b = n;
}
One layer, in outline:
for (i = tid; i < n; i += T) { rmsnorm(...); act_quant(...); } BAR;
split(qdim): Wq split(kvdim): Wk, Wv BAR;
for (i = tid; ...) { bias; q/k norm; rope; write K,V to cache } BAR;
for (row, head pairs = tid; ...) attention BAR;
quantize attention output BAR;
split(dim): Wo BAR;
residual; rmsnorm; quantize BAR;
split(ffn): Wgate, Wup BAR;
silu(gate) * up; quantize BAR;
split(dim): Wdown BAR;
residual // same row → same thread as the next layer's first step, so no barrier
That's 10–11 barriers per layer, around 300 per token for a 28-layer model. A barrier that costs even 10 µs would eat a third of the time budget at 30 tok/s, so it has to be cheap.
A sense-reversing spin barrier
Each thread flips a local "sense" bit. The last thread to arrive resets the counter and publishes the new sense; everyone else spins until they see it. No locks, no syscalls, and it's reusable immediately because the sense alternates.
static void bar_wait(barrier_t *b, int *local) {
int s = !*local;
*local = s;
if (atomic_fetch_add_explicit(&b->count, 1, memory_order_acq_rel) == b->n - 1) {
atomic_store_explicit(&b->count, 0, memory_order_relaxed);
atomic_store_explicit(&b->sense, s, memory_order_release);
} else {
unsigned spins = 0;
while (atomic_load_explicit(&b->sense, memory_order_acquire) != s) backoff(&spins);
}
}
count and sense sit on separate 64-byte cache lines so arrivals don't bounce the line the spinners are reading.
Stages talk through a lock-free queue
A stage doesn't pass tensors around. It passes a group id. Each group owns an activation buffer; a stage runs its layers on that buffer in place, then hands the id to the next stage. That's the "previous layer writes into the next layer's path" idea: there is no copy between stages, the next stage simply starts reading the buffer the previous one finished writing.
The handoff is a single-producer, single-consumer ring of ints:
static void q_push(spsc_t *q, int v) {
uint32_t t = atomic_load_explicit(&q->tail, memory_order_relaxed);
q->buf[t % QCAP] = v;
atomic_store_explicit(&q->tail, t + 1, memory_order_release);
}
static int q_pop(spsc_t *q) {
uint32_t h = atomic_load_explicit(&q->head, memory_order_relaxed);
unsigned spins = 0;
while (atomic_load_explicit(&q->tail, memory_order_acquire) == h) backoff(&spins);
int v = q->buf[h % QCAP];
atomic_store_explicit(&q->head, h + 1, memory_order_release);
return v;
}
The release store on tail and the acquire load in q_pop are what make the producer's writes to the activation buffer (and to the stream state) visible to the consumer. Only the stage leader (thread 0) touches the queue; it then broadcasts the group id to its team through a barrier.
The last stage samples the next token, writes it into the stream state, and pushes the group back to stage 0. With G ≥ S groups in flight, every stage always has something to do: while CCD 1 runs the second half of the layers for group A, CCD 0 runs the first half for group B.
Weights live where they're used
Each stage's threads load their own layers from the memory-mapped file. That copy is the first write to those pages, so on a NUMA system the memory lands on the right node ("first touch"). This machine has a single NUMA node, so the more useful part is the allocation itself: weight buffers are 2 MB aligned and marked with madvise(MADV_HUGEPAGE). Streaming 2 GB of weights through 4 KB pages means a lot of TLB misses; huge pages cut that down.
Splitting the layers
Stages should take equal time, and layers aren't the only cost. The 151,936-token lm_head is about 5 layers' worth of weights for the 1.5B model, and it always lives in the last stage. The auto-split counts it, and weights each stage by its speed (core count, or a configurable factor for a GPU):
double lp = 2.0*dim*qdim + 2.0*dim*kvdim + 3.0*dim*ffn; // weights per layer
double total = n_layer + (double)vocab * dim / lp; // + lm_head, in "layers"
// cumulative share of total, proportional to each stage's speed
For two equal CPU stages that puts layers 0–15 on CCD 0 and 16–27 plus the head on CCD 1.
The pipeline result
| Config | tok/s total | per stream |
|---|---|---|
| 1 stage, 12 cores, 1 stream | 30.5 | 30.5 |
| 2 stages (one per CCD), 2 groups × 1 | 32.3 | 16.2 |
| 1 stage, batch 8 | 138.7 | 17.3 |
| 2 stages, 2 groups × 8 | 143.8 | 9.0 |
The pipeline matches the single 12-core team. It doesn't beat it. The reason is Part 1 again. One conversation is strictly sequential: token t+1 can't start until token t has passed through every layer. In a two-stage pipeline with one stream, one CCD is always idle. With two groups both CCDs work, but each CCD now has half the cores per token, and total throughput is still capped by the same memory bandwidth.
So for this machine and one user, a pipeline is the wrong tool; one team using all 12 cores is just as fast and simpler. The pipeline structure still earned its place: it is exactly what lets a GPU stage and a CPU stage work at the same time in Part 5, and the group/queue machinery became the server's request scheduler in Part 7.
Rows: the change that mattered most
The first version processed a prompt one token at a time, like decoding. At about 30 tok/s, a 685-token prompt would take over 20 seconds before the first output token. But prompt tokens are all known in advance, so they can go through each layer as a batch, sharing every weight load.
The fix was to stop thinking in "streams" and start thinking in rows. A row is a (stream id, position, token). Each pipeline step processes a list of rows:
typedef struct {
int n, nw;
int *sid, *pos, *tok; // [rmax] one entry per row
int *want, *wb; // rows that need logits, and which stream they belong to
int *cnt; // rows per stream this step
float *x; // [rmax][dim] activations, handed from stage to stage
} work_t;
A decoding stream contributes one row. A stream that's still reading its prompt contributes a chunk of up to 64 rows at consecutive positions. Only the last row of a finished prompt needs logits, so the expensive lm_head runs only on want rows.
Why is causal attention still correct with many rows of the same stream in one batch? Because every row writes its K and V into the cache at its own position before the barrier, and attention runs after it. Row at position p then attends to cache positions 0..p, which are all filled, including the rows from this same chunk. No masks needed; the barrier is the mask.
| Prompt processing | tok/s |
|---|---|
| Token by token (before) | about 30 |
| Batched rows, CPU | 434 |
| Batched rows, GPU (Part 5) | 1,306 |
A 14× improvement from changing the shape of the work, with no kernel changes.
The row abstraction paid off twice more: speculative verification (Part 6) is just "k+1 rows of one stream", and KV cache reuse (Part 7) is just "start the prompt rows at a later position".
Sampling inside the team
The lm_head produces 151,936 logits per row. Scanning them for the maximum on one thread costs about 3% of a batch-32 step. Since each thread already computed its own slice of the vocabulary, it finds the best token in its slice right after its matmul, and the leader only compares T candidates:
for (int j = 0; j < nw; j++) {
const float *lg = S->logits + (size_t)j * V;
int bi = a;
for (int i = a + 1; i < b; i++) if (lg[i] > lg[bi]) bi = i;
S->best_i[tid * e->B + j] = bi;
S->best_v[tid * e->B + j] = a < b ? lg[bi] : -INFINITY;
}
Rows with temperature above zero still go through full top-k sampling on the leader.
Spinning politely
Spin-waiting is what makes the barrier and queue fast, but it has a cost: when the server is idle, every pinned thread would spin at 100% forever. qpipe spins for about 32K iterations, then sleeps in 50 µs naps:
static inline void backoff(unsigned *spins) {
if (++*spins < (1u << 15)) _mm_pause();
else { struct timespec ts = {0, 50000}; nanosleep(&ts, NULL); }
}
Under load, waits are short and never reach the nap. When idle, the server process sits at 0% CPU. In server mode, stage 0 also parks groups that have no active request and sleeps on a condition variable until a request arrives.