A fast engine is only useful if other programs can talk to it. This last part turns qpipe into a server: an OpenAI-compatible HTTP API written in plain C, a small chat UI, tool calling in Qwen's exact format, and KV cache reuse across requests, which halved the time of a two-turn agent loop.
A small HTTP server
The server is under 900 lines of C, JSON parser included, with no dependencies: one thread per connection, Connection: close, and a tiny JSON parser. Endpoints:
| Endpoint | What it does |
|---|---|
POST /v1/chat/completions |
OpenAI chat API: streaming (SSE) or plain JSON, temperature, top_k, max_tokens, tools |
GET /v1/models |
model id and context length |
GET /stats |
active slots, token counts, per-stage busy time, speculative acceptance |
GET / |
the chat UI, a single HTML file |
Because it speaks the OpenAI format, existing clients work by changing one line: base_url="http://127.0.0.1:8077/v1".
Defaults are conservative: it binds to 127.0.0.1, caps headers at 64 KB and bodies at 4 MB, times out slow clients after 30 seconds, and serves only the one UI file (no path handling, so no path traversal).
From requests to engine slots
The engine from Part 4 runs a fixed number of slots (groups × streams per group). The HTTP thread and the pipeline never share a lock on the hot path:
- The HTTP thread claims a free slot with a compare-and-swap on its
busyflag, then publishes the request with an atomic pointer store. - Stage 0 picks the request up the next time that slot's group passes through. Only stage 0 initializes a stream, and only while its group isn't in flight, so there's no race with the other stages.
- The last stage appends each token to the request under the request's own mutex and signals its condition variable. The HTTP thread wakes, decodes and streams.
- When the stream ends, the engine marks the request done last, after which it never touches it again, so the HTTP thread can free it safely.
static void stream_end(eng_t *e, stream_t *s, int stopped) {
req_t *r = s->req;
s->req = NULL;
s->done = 1;
s->kvn = s->pos; // remember how much KV is valid (for reuse, below)
s->dkvn = s->pos - (s->dlag >= 0 ? 1 : 0);
pthread_mutex_lock(&r->mu);
r->stopped = stopped; r->t_end = now(); r->done = 1;
pthread_cond_broadcast(&r->cv);
pthread_mutex_unlock(&r->mu); // r may be freed by the HTTP thread after this
atomic_store_explicit(&s->busy, 0, memory_order_release);
}
If all slots are busy, the request waits for one (up to two minutes, then a 503). Groups with no active request are parked by stage 0 instead of circulating, and when every group is parked, stage 0 sleeps on a condition variable until a request arrives. With the idle backoff from Part 4, an idle server uses 0% CPU.
If the client disconnects mid-answer, the next failed send() sets a cancel flag, the engine ends the stream at its next step, and the slot is free again.
Tested: 8 concurrent requests on 4 slots finished in 0.43 s, all correct.
Streaming without breaking UTF-8
Tokens are byte sequences, and a single character can be split across tokens. Hindi characters are 3 bytes, emoji 4, and Qwen's byte-level BPE happily splits them. If each token's bytes are sent as soon as they arrive, the client receives invalid UTF-8 in the middle of a JSON string.
The server holds back an incomplete sequence at the end of the buffer and sends it with the next chunk:
static size_t utf8_complete(const char *s, size_t n) {
size_t i = n, back = 0;
while (i > 0 && back < 4) {
unsigned char c = s[i - 1];
if ((c & 0xC0) != 0x80) { // found the lead byte (or ASCII)
size_t need = c < 0x80 ? 1 : (c & 0xE0) == 0xC0 ? 2 : (c & 0xF0) == 0xE0 ? 3 : 4;
return back + 1 >= need ? n : i - 1; // complete: send all; else stop before it
}
i--; back++;
}
return n;
}
A Hindi question with an emoji in the answer streamed cleanly. (The 1.5B model claimed India's capital is America, which is a model problem, not a server one.)
The chat UI
One HTML file served at /: streaming replies rendered with a small markdown subset (code blocks, inline code, lists, bold), a stop button, a settings panel (system prompt, temperature, top-k, max tokens), and a line under each reply like "328 tok · 169.7 tok/s · first token 28 ms · prompt 36 tok". The header polls /stats and shows live throughput and which device runs which layers.
Tool calling, Qwen style
OpenAI clients send tools as JSON schemas and expect tool_calls back. Qwen models are trained on a specific text format. The server translates both ways.
Into the prompt, the tools go into the system turn exactly as Qwen's chat template writes them, with each schema serialized the way Python's json.dumps does (", " and ": " separators), since that's what the model saw in training:
# Tools
You may call one or more functions to assist with the user query.
You are provided with function signatures within <tools></tools> XML tags:
<tools>
{"type": "function", "function": {"name": "get_weather", ...}}
</tools>
For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
Tool results (role: "tool") become a user turn wrapped in <tool_response> tags, with consecutive results grouped into one turn. Earlier assistant tool calls are re-rendered inside <tool_call> tags.
Out of the model, the server watches the token stream. Text tokens stream as content. When the <tool_call> token appears, it buffers until </tool_call>, parses the JSON, and emits an OpenAI tool_calls entry. If anything was called, finish_reason is "tool_calls". This works in streaming mode too.
The bug: tool calls that came back as text
The first test looked almost right. The model called both tools correctly, but the response came back as plain text containing <tool_call>... and finish_reason: "stop". The parser never fired.
The cause: <tool_call> and </tool_call> are added tokens in Qwen's vocabulary (ids 151657 and 151658). Hugging Face tokenizers turn those strings into the single token wherever they appear in the template text. My server encoded the template's <tool_call> as ordinary characters, so the model saw a character-by-character version, copied that style, and never emitted the real token my parser was watching for.
The fix is a template-only encoder that maps those four tags to their token ids. User-supplied text still never produces special tokens, which keeps the injection protection from Part 2:
static int push_tmpl(int *t, int n, int cap, const char *s) {
static const char *tags[4] = {"<tool_call>", "</tool_call>", "<tool_response>", "</tool_response>"};
int ids[4] = {TC_OPEN, TC_CLOSE, TR_OPEN, TR_CLOSE};
... // split on tags; push ids for tags, plain-text tokens for everything else
}
One more client detail: OpenAI SDKs send assistant messages that carry tool calls with "content": null. The first version rejected that with a 400; null now means empty.
An agent loop, end to end
A 40-line Python script with two tools (weather and a calculator) talking to qpipe running Qwen3-4B with the 0.6B draft:
turn 0: finish=tool_calls
-> get_weather({'city': 'Pune'}) = {"city": "Pune", "temp_c": 29, "condition": "sunny"}
-> get_weather({'city': 'Delhi'}) = {"city": "Delhi", "temp_c": 34, "condition": "sunny"}
turn 1: finish=stop
ANSWER: Delhi is hotter than Pune right now. The temperature in Delhi is 34°C, while
in Pune it is 29°C. The difference is 5 degrees Celsius.
Two parallel tool calls, a correct final answer, and speculative decoding accepting 83% of drafts on the tool-call JSON. The whole loop took 2.07 seconds. The remaining cost wasn't generation, it was the prompt.
KV cache reuse across requests
Every agent turn resends everything: system prompt, tool schemas, the conversation so far. Each turn processed 200–300 prompt tokens from scratch, about 450–780 ms on the 4B model. But almost all of it was identical to what the slot had just processed.
Each slot now remembers which token sits at each position of its KV cache, and how far the cache is valid, separately for the target and the draft:
int *kvtok, kvn, dkvn; // token at each position; target / draft KV valid for [0, kvn) / [0, dkvn)
When a request is activated, stage 0 finds the longest common prefix with the slot's previous sequence and starts prefill there instead of at position 0:
int reuse = 0, lim = s->kvn < s->dkvn ? s->kvn : s->dkvn;
while (reuse < lim && reuse < r->np - 1 && s->kvtok[reuse] == r->prompt[reuse]) reuse++;
s->pos = reuse; // prefill rows start here; earlier KV is reused as-is
Three details make this correct:
- It's capped at
np − 1. At least one prompt token must be processed, to produce the logits for the first output token. - The draft's limit counts too. Speculative decoding needs the draft's cache valid over the same prefix. The draft can lag one token behind (Part 6), so it has its own
dkvn. - Stale entries are harmless. Positions past the reuse point still hold the old conversation's K/V, but prefill overwrites them before attention ever reads them. The same reason rollback is free.
Generated tokens are recorded into kvtok as they're emitted, so an assistant's reply is part of the cached prefix for the next turn. When several slots are free, a new request goes to the slot whose cache shares the longest prefix with it. Responses report how much was reused in the OpenAI field usage.prompt_tokens_details.cached_tokens.
| cache off | cache on | |
|---|---|---|
| Agent turn 2 (send tool results): time to first token | 780 ms | 257 ms (222 of 323 tokens reused) |
| New conversation, same tools: time to first token | 478 ms | 56 ms (196 of 219 reused) |
| Whole two-turn agent loop | 1.79 s | 0.87 s |
The final answers were identical with and without the cache.
Why only 222 of 323 on the second turn? The match ends one token into the assistant's tool call. The client sends the tool call back as structured JSON, the server re-renders it into Qwen's template text, and that re-rendered text doesn't tokenize exactly the way the model generated it: whitespace grouping differs, and the server returns arguments as compact JSON while the model wrote them with spaces. Keeping the model's exact tokens for each call and re-rendering them verbatim would extend the match through the whole history. That's the next improvement.
What I'd take away
- Do the bandwidth math first. Tokens per second is bounded by bytes per token. It tells you which optimizations can work before you write them.
- Batch everything you can. Prompts, verification passes, concurrent users. The same weights read once, used many times.
- Shape the work, not just the kernels. Rows instead of streams gave batched prefill (14×), made speculative verification a few lines, and made KV reuse a one-line change to the starting position.
- Keep a determinism check. Identical greedy output across every backend caught every mistake and made aggressive changes safe.
- Respect the template. A model's chat and tool format, down to whether a tag is one token or eleven, is part of its interface.
- Small models are useful. A 4B model at 60–140 tok/s, with tools and sub-second agent turns, handles routing, extraction and tool-driven tasks well on a desktop GPU.
Final numbers
| Setup | Result |
|---|---|
| Qwen2.5-1.5B, CPU 12 cores, 1 stream | 30.5 tok/s (Q8), 50 tok/s (Q4) |
| Qwen2.5-1.5B, RTX 3060, 1 stream | 157 tok/s (Q8), 216 tok/s (Q4) |
| Qwen2.5-1.5B, CPU + GPU, 32 streams | 1,467 tok/s total (Q4) |
| Prompt processing, GPU | 1,306–1,438 tok/s |
| Qwen3-4B + 0.6B draft, code | 98 tok/s (Q8), 137 tok/s (Q4) |
| Two-turn tool-calling agent loop | 0.87 s |
All of it in about 3,500 lines of C and CUDA, on a desktop.