Series · 7 parts

Building an LLM Inference Engine from Scratch

2026-11-21 · Servloci Engineering · Discuss on LinkedIn

qpipe is an LLM inference engine written from scratch in C and CUDA, with no dependencies, built and measured on one desktop: a Ryzen 9 7900X and an RTX 3060. It runs Qwen2.5 and Qwen3 models on the CPU, the GPU or both, with speculative decoding, an OpenAI-compatible server, tool calling and KV cache reuse.

This series covers every layer with the real code and the real numbers, including the experiments that didn't work. Start with Part 1: everything else follows from one idea, that generating a token is a memory problem, not a math problem.

PART 1

The Physics of Token Speed: Why Memory Bandwidth Sets the Limit

I wanted to answer a simple question: how fast can a normal desktop run a language model if every core, every cache and the GPU are used deliberately instead of by accident? The answer turned into qpi

PART 2

Reading GGUF and Running Qwen by Hand

Before any optimization, the engine has to produce the right answer. This part covers the three pieces that turn a file on disk into text: parsing the GGUF model file, the byte-level BPE tokenizer, an

PART 3

CPU Kernels: Q8, Q4 and AVX-512 VNNI

Part 1 showed that single-stream decoding is limited by memory bandwidth. So why write fast kernels at all? Because the moment you batch, whether that's several users, a prompt being prefilled, or a s

PART 4

Threads, Pinning and the Pipeline

The original goal of qpipe was a "synced pipeline": split the model's layers into stages, give each stage its own cores, and have each layer group write straight into the next group's input. This part

PART 5

Going to the GPU: from 230 to 1,300 tok/s

The RTX 3060 has about 360 GB/s of memory bandwidth, five to six times what the CPU gets from DDR5. By Part 1's formula that's a 5–6× higher ceiling for single-stream decoding, and far more arithmetic

PART 6

Speculative Decoding: More Tokens per Weight Read

After Part 5, single-stream decoding on the GPU ran close to the memory-bandwidth ceiling. The formula from Part 1 says the only way past it is to get more than one token out of each pass over the wei

PART 7

Serving It: OpenAI API, Tool Calling and KV Reuse

A fast engine is only useful if other programs can talk to it. This last part turns qpipe into a server: an OpenAI-compatible HTTP API written in plain C, a small chat UI, tool calling in Qwen's exact