tl;dr: llama-server serving bartowski/Qwen3.8-27B-GGUF:Q6_K at 9.5 tok/s decode, with one server flag that silently breaks every request that doesn’t carry an explicit reasoning_effort.
This is the sibling of the Rapid-MLX post. Same model, same machine, same harness (Pi), different serving stack and a different quantization. The MLX run was 4-bit; this one is Q6_K, which is the quantization I actually want to live with, and llama.cpp is the server that gets me there.
The server
The whole command, as it is running:
1llama-server -hf bartowski/Qwen3.8-27B-GGUF:Q6_K \
2 --port 8080 \
3 -ngl 99 \
4 -fa on \
5 -c 32768 \
6 -b 2048 -ub 2048 \
7 -t 10 \
8 --parallel 1 \
9 --cache-type-k q8_0 \
10 --cache-type-v q8_0 \
11 --load-mode mlock \
12 --jinja \
13 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 \
14 --reasoning-effort high \
15 --reasoning-preserve
Worth noting flag by flag, because three of them are load-bearing and one is a trap:
-hfpulls the repo straight from Hugging Face and caches it locally. No manual download, and the right file is picked from the repo’s file list.-ngl 99offloads every layer to the GPU. There is no scenario on this machine where you would not do that.-fa onenables flash attention. On a hybrid model this also interacts with the KV layout, so leave it on.-c 32768is the context. The model was trained on 262144, and the GGUF is happy to go much further, but 32k is what the harness is configured for and it is what fits comfortably.-b 2048 -ub 2048sets the prompt processing and update batch sizes. Prefill gets chunked at 2048 tokens, which is what keeps prefill memory bounded on a laptop.-t 10is CPU threads, matching the ten performance cores. Most of the CPU work here is the embedding and output heads plus anything the GPU does not take.--parallel 1— one slot. I am one person. Extra slots just split the KV cache and buy nothing.--cache-type-k q8_0 --cache-type-v q8_0quantizes the KV cache to 8-bit. Same reasoning as the MLX post: at 32k context the unquantized cache is the thing that pushes a 48GB machine into unified-memory pressure territory, and 8-bit KV is the standard, well-tested way to halve it.--load-mode mlocklocks the weights in RAM so they cannot be paged or compressed out from under you mid-generation.--jinjamakes the server render the model’s own chat template instead of llama.cpp’s built-in approximation. For a model with a non-trivial reasoning template, this is not optional.--reasoning-preservekeeps the reasoning content in the rendered context across turns rather than dropping it.
The sampling flags (--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0) are Qwen’s recommended thinking-mode values. They are duplicated in the harness config below, which is redundant but harmless — the client values win, and they are identical.
The flag that breaks everything
--reasoning-effort high.
The model’s chat template does not accept high. It accepts xhigh, medium, and low, with xhigh as the default. So the server starts up fine, reports healthy, lists the model, and then fails on any request that does not carry its own reasoning_effort:
1Jinja Exception: Unexpected reasoning effort high. Supported types are
2xhigh (default), medium, and low.
HTTP 500, every time. The server never logs anything at startup that suggests the flag is invalid, because the flag is only validated when the template renders it.
The fix is to either drop the flag (the template default is xhigh) or set one the template accepts. I would set it explicitly rather than rely on the default, for the same reason I would rather see the parser flags in the command than discover a version bump changed the heuristic.
There is a second-order version of this trap in the harness config, below. It is worth understanding before it bites.
The harness config
1{
2 "providers": {
3 "llama-cpp": {
4 "baseUrl": "http://127.0.0.1:8080/v1",
5 "api": "openai-completions",
6 "apiKey": "none",
7 "models": [
8 {
9 "id": "bartowski/Qwen3.8-27B-GGUF:Q6_K",
10 "name": "Qwen 27B Local (Q6_K)",
11 "reasoning": true,
12 "input": ["text", "image"],
13 "contextWindow": 32768,
14 "maxTokens": 8192,
15 "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 },
16 "thinkingLevelMap": {
17 "minimal": "low",
18 "low": "low",
19 "medium": "medium",
20 "high": "medium",
21 "xhigh": null,
22 "max": null
23 },
24 "samplingParams": {
25 "temperature": 0.6,
26 "top_p": 0.95,
27 "top_k": 20,
28 "min_p": 0
29 },
30 "compat": {
31 "supportsDeveloperRole": false,
32 "thinkingTokenBudgetField": "thinking_budget_tokens"
33 }
34 }
35 ]
36 }
37 }
38}
A few deliberate choices in there:
thinkingLevelMap is the bridge between the harness’s thinking levels and the model’s reasoning_effort vocabulary. The harness has more levels than the model has efforts, so the map collapses them: minimal and low both land on low, medium and high both land on medium. The model’s top effort is xhigh, and the map does not use it — xhigh and max map to null.
That null is the second-order trap. A null entry means the harness does not send the field at all, and a missing field means the server falls back to its own default. With --reasoning-effort high on the command line, that fallback is the value the template rejects, so the two highest thinking levels are the ones that 500. The map and the server flag have to be read as a pair, and neither one is wrong in isolation.
maxTokens: 8192 is double what the MLX config had, and that doubling was the single most important fix in that post. A reasoning model that gets truncated mid-thought repeats itself from the top of the budget. 8192 is not generous, but it is not the thing killing turns.
contextWindow: 32768 matches the server’s -c exactly. The harness will not ask for more than the server will give, and asking for more is how you find out the hard way.
supportsDeveloperRole: false keeps the harness from sending a developer-role message, which this template does not map.
What it costs in tokens per second
The model is 27.3B parameters. Q6_K puts the weights at 23.45GB on disk, and the running process sits at 28.3GB RSS out of 48GB total, which leaves the OS and everything else a comfortable 19GB.
Decode is memory-bandwidth-bound, so the ceiling is arithmetic: the M4 Pro does 273GB/s, the weights are 23.45GB, so the theoretical maximum is about 11.6 tok/s.
Measured, with a 51-token prompt and 128 tokens generated:
1prompt=51 gen=128 prefill=53.4 t/s gen=9.5 t/s
Nine point five. That is about 82% of the bandwidth ceiling, which is the same efficiency the MLX build was getting. There is no tuning left in this number; it is set by the quantization and the memory bus.
Prefill, measured with a 6040-token prompt:
1prompt=6040 gen=64 prefill=100.5 t/s gen=8.8 t/s wall=67.0s
A hundred tokens per second prefill, and it scales linearly with prompt length, which means a 16k conversation is roughly two minutes of waiting before the first token. Same story as the MLX run, same structural cause: the harness inserts tool results into the middle of the conversation, so prefix caching never gets a clean leading match and every turn re-prefills almost the whole thing.
Q6_K against 4-bit
The MLX post measured 15.09 tok/s on the 4-bit build. This is 9.5. The 4-bit weights are 15GB; Q6_K is 23.45GB. Bandwidth-bound decode means the speed ratio is basically the inverse of the size ratio, and 15/23.45 predicts 9.9 against the measured 9.5. The quantization bought quality at exactly the price the physics said it would.
Whether that trade is right depends on what you are doing. For short, well-specified tasks the 4-bit model is faster and good enough. For the longer agentic sessions where the model has to hold a role, a set of constraints, and a multi-step plan in its head, the extra bits are the difference between a model that follows the CLAUDE.md and one that drifts out of it. I have been running the Q6_K build for the longer sessions and the 4-bit for quick questions.
Where it landed
The one change from the command above since I started writing this: --reasoning-effort high is going, replaced by --reasoning-effort medium, with the understanding that the harness map already collapses its own high level to medium and that the null entries at the top of the map will now fall back to a value the template accepts instead of one it rejects.
Everything else is standing. The server is unremarkable in the best way: it starts, it serves, the numbers match the theory, and the only thing that bit me was a flag the server should have refused at startup.