qpipe is an LLM inference engine written from scratch in C and CUDA, with no dependencies, built and measured on one desktop: a Ryzen 9 7900X and an RTX 3060. It runs Qwen2.5 and Qwen3 models on the CPU, the GPU or both, with speculative decoding, an OpenAI-compatible server, tool calling and KV cache reuse.
This series covers every layer with the real code and the real numbers, including the experiments that didn't work. Start with Part 1: everything else follows from one idea, that generating a token is a memory problem, not a math problem.
PART 1The Physics of Token Speed: Why Memory Bandwidth Sets the Limit
I wanted to answer a simple question: how fast can a normal desktop run a language model if every core, every cache and the GPU are used deliberately instead of by accident? The answer turned into qpi
PART 2Reading GGUF and Running Qwen by Hand
Before any optimization, the engine has to produce the right answer. This part covers the three pieces that turn a file on disk into text: parsing the GGUF model file, the byte-level BPE tokenizer, an
PART 3CPU Kernels: Q8, Q4 and AVX-512 VNNI
Part 1 showed that single-stream decoding is limited by memory bandwidth. So why write fast kernels at all? Because the moment you batch, whether that's several users, a prompt being prefilled, or a s
PART 4Threads, Pinning and the Pipeline
The original goal of qpipe was a "synced pipeline": split the model's layers into stages, give each stage its own cores, and have each layer group write straight into the next group's input. This part
PART 5Going to the GPU: from 230 to 1,300 tok/s
The RTX 3060 has about 360 GB/s of memory bandwidth, five to six times what the CPU gets from DDR5. By Part 1's formula that's a 5–6× higher ceiling for single-stream decoding, and far more arithmetic
PART 6Speculative Decoding: More Tokens per Weight Read
After Part 5, single-stream decoding on the GPU ran close to the memory-bandwidth ceiling. The formula from Part 1 says the only way past it is to get more than one token out of each pass over the wei
PART 7Serving It: OpenAI API, Tool Calling and KV Reuse
A fast engine is only useful if other programs can talk to it. This last part turns qpipe into a server: an OpenAI-compatible HTTP API written in plain C, a small chat UI, tool calling in Qwen's exact