tl;dr: Getting Qwen 3.8 27B to stop looping under Pi.
How I got here
I have been circling this for a while. ChatGPT landed in 2022 and I did what everyone did: chat UI, then code completion in VS Code, then a bit of Gemini, and I did not think much of any of it. Then Claude Code started making headlines, I paid for it, I threw some real projects at it, and it clicked. It has been my daily driver since.
Somewhere in there I bought a MacBook Pro with 48GB of RAM, which is enough unified memory to be dangerous, and started dabbling with local models. Gemma 4 32B, Qwen 3.6, wired into Zed and into Claude Code. I could never get any of them to hold together for more than a few turns before they started going in circles, and after enough evenings of that I put it down and went back to Claude Code where I had a rhythm going.
Qwen 3.8 27B landing a few days ago was enough to get me curious again, so I gave it a spin with Rapid-MLX serving and Pi as the harness, and predictably enough it went in circles too. It looked promising in between the circling though, which is the annoying bit, so rather than give up a second time I decided to actually measure the thing and read the logs properly (with Claude and Gemini doing a lot of the heavy lifting on the troubleshooting, it should be said).
The config I started with
Here is the setup I was handed. It looks right. It reads like someone who knew what they were doing wrote it.
1rapid-mlx serve mlx-community/Qwen3.8-27B-4bit \
2 --port 8000 \
3 --continuous-batching \
4 --reasoning-parser qwen3 \
5 --tool-call-parser hermes
1{
2 "providers": {
3 "rapid-mlx": {
4 "baseUrl": "http://127.0.0.1:8000/v1",
5 "apiKey": "none",
6 "api": "openai-completions",
7 "models": [
8 {
9 "id": "mlx-community/Qwen3.8-27B-4bit",
10 "name": "Qwen 3.8 27B (MLX)",
11 "contextWindow": 32768,
12 "maxTokens": 4096,
13 "supportsTools": true,
14 "temperature": 0.65,
15 "topP": 0.9,
16 "stop": ["<|im_end|>", "<|endoftext|>", "</think>"]
17 }
18 ]
19 }
20 }
21}
The server line is fine. The JSON is where the damage is, and almost none of it is visible from reading it.
Silently ignored
Pi’s model config has a fixed set of fields: id, name, api, reasoning, thinkingLevelMap, input, contextWindow, maxTokens, samplingParams, cost, compat.
Count what is in the block above that is not on that list: temperature, topP, stop, supportsTools. All four are quietly dropped, with no warning and no log line to tell you about it, so the file parses and the agent runs and the sampling settings you sat there tuning never actually leave the machine.
Sampling goes in samplingParams, which is merged verbatim into the request body. Verbatim is the important word: the keys have to be the snake_case names the API expects, so top_p and not topP. It also means server-specific keys work, which is how you get top_k and min_p through a harness that has no concept of either.
NOTE: only OpenAI-compatible APIs apply samplingParams. On openai-completions you are fine. On other APIs it is ignored, which would put you right back where you started.
Stop sequences
"stop": ["</think>"] is the one that made me laugh once I saw it.
The reasoning parser already separates thinking from the response. Adding </think> as a stop sequence tells the server to halt generation at exactly the moment the model finishes reasoning and is about to answer. You get the thinking. You never get the answer.
Delete the whole array. You do not need stop sequences here at all. The parser handles the thinking boundary and the tokenizer handles <|im_end|>.
The actual loop
Qwen 3.8 supports reasoning_effort, and the default is xhigh. The model card is not shy about it: extra-high thinking, on by default, for complex tasks demanding thorough analysis.
Now put that against maxTokens: 4096.
I ran a deliberately trivial request through the server to watch it happen. Count from 1 to 400, one number per line.
1[schedule] prompt_tokens=67 tokens_to_prefill=67 max_tokens=4096
2[Metal memory] active=16.1GB peak=16.2GB step=4096
3[cache_store] tokens=4163 (67 prompt + 4096 output) stored=True
Four thousand and ninety six output tokens, which is to say it hit the ceiling exactly, and it never finished counting. It spent the entire budget thinking about counting.
That is the loop, and it is entirely self-inflicted. The turn truncates mid-thought, the harness compacts what it has, the model picks the same reasoning up from the start, and from the outside it looks for all the world like a model too stupid to make progress on a task a first-year could do.
Pi’s own default is 16384. The config I was handed had lowered it.
Parsers
I spent a while convinced the tool parser was wrong. vLLM’s recipes for this model family use qwen3_coder, so I went looking for it in Rapid-MLX.
It is not there, and it would have been wrong anyway. The qwen3_coder parser only activates on models whose chat template contains <tool_call>\n<function=, and the regular Qwen 3.x templates do not. That parser is for the Coder checkpoint, which this is not, despite this very much being a coding model.
Rapid-MLX ships hermes, which is correct, and it does not need telling:
1Auto-detected model family 'qwen3' -> tool_call_parser=hermes,
2 reasoning_parser=qwen3, is_hybrid=False, supports_spec_decode=True
It also loads the model’s own sampling defaults without being asked:
1generation_config: loaded sampling defaults from generation_config.json:
2 {'temperature': 1.0, 'top_p': 0.95, 'top_k': 20}
Those are the values Qwen recommends for thinking mode, so the temperature: 0.65 I had been carefully setting was both wrong and being thrown away, and the model was running on sensible defaults the whole time by pure luck.
MTP, disabled at runtime
Qwen 3.8 ships a multi-token prediction head and MTP is the single biggest decode win available on a memory-bandwidth-limited machine, so this was the first thing I went looking for.
The startup log answers it in two lines:
1Auto-detected model family 'qwen3' -> ... supports_spec_decode=True
2Runtime probe: model has ArraysCache layers - marking as hybrid, disabling spec decode
3Model profile: -> hybrid (linear-attention/Mamba), throttle ON, spec decode OFF
The family heuristic said yes. The runtime probe looked at the actual loaded weights, found GatedDeltaNet layers that cannot be rolled back for speculative verification, and turned it off. Correctly, as far as I can tell.
So no MTP on this model on this stack, no matter what flag you pass. Worth knowing before you go reading benchmark numbers from people running the GGUF build through llama.cpp, where self-speculation does work.
Measuring it properly
This is the bit where I had been quietly fooling myself for a while.
1rapid-mlx bench mlx-community/Qwen3.8-27B-4bit --max-tokens 512
2 Tokens/second: 28.28
Twenty eight. Not bad for a 27B on a laptop. Except the bench defaults to ten concurrent prompts, and with ten streams in flight the weight reads amortise across all of them. That number describes throughput under load. I am one person typing at an agent.
1rapid-mlx bench mlx-community/Qwen3.8-27B-4bit --num-prompts 1 --max-tokens 512
2 Tokens/second: 15.09
Fifteen. That is the real one, and it is close to the theoretical ceiling for this hardware. The M4 Pro does 273 GB/s and the weights are 15GB on disk, which puts the bandwidth-bound limit somewhere around 18 tok/s. MLX is getting about 83% of that. There is no tuning left in this number.
Then through the actual server, generating 4096 tokens:
1Chat completion: 4096 tokens in 276.29s (14.8 tok/s)
And through Pi, on short agent turns:
1Chat completion (stream): 44 tokens in 16.82s (2.6 tok/s)
Same model, same machine, same afternoon. 28.28, 15.09, 14.8, 2.6. Every one of those numbers is correct and they measure four different things. The last one is not slow decode, it is a 14 second prefill with a 44 token response hung off the end of it.
KV cache quantization
With contextWindow set to 131072 I wanted the KV cache smaller, so: does --kv-cache-quantization --kv-cache-quantization-bits 8 actually do anything on a hybrid model?
Two servers, identical prompt, cache cleared between them.
1with quantization: tokens=4163, cache_mem=772MB, 4096 tokens in 294s
2without quantization: tokens=4163, cache_mem=900MB, 4096 tokens in 276s
128MB apart, which looks underwhelming until you split fixed cost from per-token cost. If quantization halves the per-token KV, then that entry was 256MB unquantized and 128MB quantized, leaving 644MB of fixed cost identical in both runs. That fixed part is the DeltaNet RNN state, which does not quantize, and it matches the per-entry figures in the startup log.
So: roughly 61KB per token unquantized, 31KB quantized. At 131k context that is 8.1GB against 4.0GB.
Empirically (and with one run each, so treat the timing as indicative), it costs about 6% throughput.
I kept it, and not for the memory headroom on a good day. The warning at startup is the reason:
Apple Silicon firmware can panic the whole system rather than raise an OOM error when unified-memory pressure exceeds the iBoot AMCC threshold.
My first benchmark ran with the OS already holding 26.5GB. Weights plus 8GB of unquantized KV plus that is 49.5GB on a 48GB machine. Quantized it lands at 45.5GB. Trading 6% throughput against a kernel panic that takes the whole session with it is not a close call.
Cache misses
The one thing I could not fix, and the one that ends up mattering most.
1prompt_tokens=2636 -> first token after 21.6s
2prompt_tokens=3284 -> first token after 26.9s
3prompt_tokens=6949 -> first token after 58.9s
4prompt_tokens=8132 -> first token after 68.3s
Every one of those is a MISS. Prefill runs at a flat 120 tok/s or so and scales linearly with the prompt, which means a 16k conversation is a little over two minutes of waiting before the first token appears, and 32k is four and a half.
Each turn shares almost all of its prefix with the one before it and re-prefills the lot anyway. I tried giving the cache more room, and I tried pulling samplingParams back out in case a sampler mismatch was invalidating the match, and neither made any difference.
The reason is structural: Pi inserts the tool result into the middle of the conversation, so turn two is not turn one plus tokens on the end. Prefix caching needs an exact leading match and it does not have one.
Decode sits around 13 to 15 tok/s once you subtract the prefill, so on any session long enough to be useful the waiting is mostly the machine re-reading a conversation it finished writing thirty seconds earlier. It is also the part that spins the fans, prefill being dense compute across the whole prompt where decode is memory-bound and comparatively idle.
NOTE: I have not chased this into the Rapid-MLX source. It may be solvable. If you have solved it, I would like to know.
Where it landed
1rapid-mlx serve mlx-community/Qwen3.8-27B-4bit \
2 --port 8000 \
3 --reasoning-parser qwen3 \
4 --tool-call-parser hermes \
5 --enable-auto-tool-choice \
6 --kv-cache-quantization \
7 --kv-cache-quantization-bits 8
1{
2 "providers": {
3 "rapid-mlx": {
4 "baseUrl": "http://127.0.0.1:8000/v1",
5 "apiKey": "none",
6 "api": "openai-completions",
7 "compat": {
8 "supportsDeveloperRole": false,
9 "supportsReasoningEffort": false,
10 "thinkingFormat": "qwen-chat-template"
11 },
12 "models": [
13 {
14 "id": "mlx-community/Qwen3.8-27B-4bit",
15 "name": "Qwen 3.8 27B (MLX)",
16 "reasoning": true,
17 "input": ["text", "image"],
18 "contextWindow": 131072,
19 "maxTokens": 32768
20 }
21 ]
22 }
23 }
24}
The parsers are redundant, since the server auto-detects both, but I would rather see them in the command than find out later that a version bump changed the heuristic.
supportsReasoningEffort: false means Pi never sends the field, so the model runs at its own xhigh default. Given I care about the answer being right more than about it arriving quickly, that is the setting I would have picked anyway.
A longer session
The vowels-and-pytest thing I had been poking at is small enough to fit in three turns, so I pointed it at something with a bit more context in it: a repo of Python katas with a CLAUDE.md in the working directory setting out how the agent should behave. I had run the same task yesterday on the old config and it went round in circles until I killed it.
Today it got through it. Four parallel tool calls on the first turn to read the kata, two markdown files and the git log, then a git log and git diff on one specific file, then a reply. Four turns, no repeated calls, and it stayed inside the role the CLAUDE.md set for it rather than wandering off and writing code it had been told not to write.
The numbers are the interesting part:
1turn 1: 2636 prompt -> 21.6s prefill, 251 tokens in 38.7s
2turn 2: 3284 prompt -> 26.9s prefill, 301 tokens in 47.8s
3turn 3: 6949 prompt -> 58.9s prefill, 600 tokens in 101.4s
4turn 4: 8132 prompt -> 68.3s prefill, 779 tokens in 127.5s
Peak Metal memory 25.2GB, prefix cache up at 5338MB and still missing on every turn. Roughly four minutes of wall clock in total, and the fans were audible for the first time all day.
Repetition, still there
I should be honest about the thinking trace on that run, because this paragraph appears in it twice, word for word:
I need to be careful about the ritual here, there’s a rule about redoing a kata from two days ago, but that one doesn’t exist yet, so I’ll just move forward with helping them verify their solution through tests.
And “I need to be careful about the ritual here” opens five separate paragraphs of reasoning.
So the circling is still in there. What changed is that with 32768 tokens of headroom it can spin for a few hundred tokens and still come out the other side, where at 4096 the same spin swallowed the whole budget and the turn died mid-thought. The ceiling was never causing the repetition, it was turning a recoverable wobble into a dead turn.
Which puts presence_penalty back on the list, Qwen documenting 0 to 2 for exactly this symptom. I had left it at 0 on the grounds that the loops were structural rather than a sampling problem, which was half right at best.
What it costs
Every turn re-prefills the whole conversation at around 120 tok/s, so the longer the session runs the worse it gets, and none of that is recoverable through config as far as I can tell. Compacting early and starting fresh sessions matters here in a way it never did for me against a hosted model.
I will stop the note here. A 6-bit build is the next thing I want to try, if one turns up, and presence_penalty before that.