Why we now serve Qwen3.6 35B-A3B instead of Qwen3.8 27B

For four days CARouter Cloud served Qwen3.8 27B, and the three posts before this one are the story of making it fast: a better draft model, then better settings for that draft, which together took it from 20.8 to 34.8 tokens a second per stream. It was not enough. A dense 27B on a box with 273 GB/s of memory bandwidth has to read all 23.4 GB of itself to produce one token, and no amount of tuning changes that arithmetic. Today the coding endpoint runs Qwen3.6 35B-A3B instead - an older model, one that scores lower on the public coding leaderboards, and a bigger one at 35B against 27B. It is also between one and a half and two times faster at everything we measured, and we could not find the quality we were supposed to be giving up.

The previous article: tuning speculative decoding on the same box, two thirds faster coding answers.

The short version

• The problem. A dense model reads every one of its parameters for every token it writes. On the GB10's 273 GB/s that put a hard ceiling of about 12 tokens a second on Qwen3.8 27B before any speculation, and the practical consequence was queueing: on an automated agentic platform, a routine coding build took an hour.
• The fix, which sounds wrong. A bigger model. Qwen3.6 35B-A3B has 35B parameters but activates only 3B of them per token, so it reads 2.0 GB per forward pass instead of 23.4. It occupies the same 21.8 GiB on disk.
• The throughput. 62.2 tokens a second single-stream against 34.8, and 367 against 241 with sixteen streams in flight - between 1.5x and 1.9x at every concurrency we measured. Prefill is 3.2x: about 6,600 tokens a second against 2,088 on a 12K prompt. The same build that took an hour takes ten minutes.
• The concurrency. From the same 36 GiB of KV budget it holds 1,837,064 tokens against 813,427 - 7.0 full 262K-token sessions at once against 3.1. Only 10 of its 40 layers keep a cache that grows with the conversation; the other 30 keep a constant.
• The quality we expected to lose. The public leaderboards put it about ten points behind on LiveCodeBench v6. On the fifty BigCodeBench problems we run ourselves, paired on the same problems on the same box, it scored 33/50 against the 27B's 32/50 - a difference of one problem, with an exact McNemar p of 1.000. On long-context retrieval it was better: 10/10 against 9/10, and 10/10 against 8/10 at 220K tokens.
• The honest caveat. Fifty problems cannot detect a ten-point gap; that would take about 218. So the claim is not that the two models are equally intelligent - it is that we looked for the difference on our own workload and could not find it, while the speed difference was impossible to miss.

Why a dense model runs out of road

Generating a token means reading the model's weights out of memory and multiplying. For a dense model that is all of the weights, every single time. Qwen3.8 27B in NVFP4 is 23.4 GB of them, and the GB10 moves about 273 GB/s, so the fastest it can possibly write is 273 divided by 23.4 - about 12 tokens a second. Everything the previous two posts achieved was speculative decoding working around that number rather than changing it: a draft model guesses several tokens ahead, the big model checks them in one pass, and the accepted ones come out at once. It got us to 34.8 tokens a second, which is roughly three accepted tokens per 12-per-second step.

A sparse mixture-of-experts model breaks the arithmetic instead of working around it. Qwen3.6 35B-A3B holds 256 experts and routes each token to 8 of them, so although all 35B parameters sit in memory, only about 3B are read per forward pass - 2.0 GB instead of 23.4. The ceiling moves from 12 tokens a second to about 140. The model is larger and the file on disk is the same size; what changed is how much of it has to be touched to produce a word.

What we measured

Both models on the same box, same 36 GiB KV budget, same 16-sequence limit, one swapped for the other because the two do not fit together. Aggregate decode across concurrent streams:

Two details make that chart understate the case. The 35B-A3B ran with no speculative decoding at all, while the 27B had the seven-token DFlash2 draft the last post shipped, landing about 3.9 accepted tokens per step. And its experts ran on vLLM's Marlin kernel, which unpacks the 4-bit weights and multiplies them in 16-bit, where the 27B gets FlashInfer's CUTLASS path. That second one is a property of the checkpoint rather than of the box: NVIDIA quantizes this model W4A16, four-bit weights against sixteen-bit activations, and the CUTLASS kernels want the activations in FP4 too, along with calibration scales a W4A16 checkpoint never carries. So these are floor numbers on both counts.

Memory is the other half, and on a shared-memory box it is the half that decides how many people you can serve at once:

The concurrency figure comes from the engine's own boot log: 1,837,064 cached tokens against 813,427 from an identical budget. The reason is architectural. Only 10 of the 35B-A3B's 40 layers use the kind of attention whose cache grows with the length of the conversation; the other 30 are linear-attention layers that keep a fixed-size state per session no matter how long it runs. The dense 27B keeps a growing cache on 16 of its 64 layers, and each of those costs more per token. Twice the sessions, from the same memory, is not a tuning win - it is what the layer layout gives you.

The part we expected to hurt

Qwen3.6 is an older generation than Qwen3.8, and the published coding benchmarks say so: LiveCodeBench v6 puts the sparse model at 80.4 against the dense model's 90.3. Ten points is not a rounding error, and we did not want to discover it in production.

So we ran the same fifty BigCodeBench problems against both, on the same box, in the same Docker sandbox, uncapped, with identical settings. Then we did something we should have done sooner: we re-measured the 27B rather than trusting its score from four days earlier. That old number had been collected under a different draft, a different vLLM version, and a run that the harness's own timeout bug had broken into pieces and restarted. Re-run cleanly, the dense model scored 32/50 where it had scored 33/50 before.

Paired on the same fifty problems, the two models agree on 43 of them - 29 both solved, 14 both failed. The sparse model alone solves four, the dense model alone solves three, and an exact McNemar test on those seven disagreements returns a two-sided p of 1.000. That is the flattest result a paired test can give.

Long-context retrieval went the other way entirely. We hide a fact in a haystack of up to 220,000 tokens and ask the model to find it, at three depths for each of four lengths. The dense model scores 9 out of 10 overall and drops to 8 at the longest setting. The sparse model scores 10 out of 10, at every length and every depth. That was the result most likely to disqualify it - we sell a 262,144-token context, and a model that answers quickly but loses the thread deep in a long file would be a regression no coding score would reveal - and it is the one that came back strongest.

It also reached the same score using far fewer words: 236,949 output tokens for the fifty problems against 558,275, with a median sample of 4,773 tokens against 9,142. The dense model hit our 32,768-token ceiling on at least one problem; the sparse model never passed 12,253. Both models were reasoning - the sparse one logged 219,951 reasoning tokens across all fifty samples - so this is brevity rather than a thinking mode left switched off. For an endpoint that bills per token and holds a context per session, arriving at the same answer in less than half the words is worth as much as the tokens-per-second.

What we cannot tell you

Fifty problems is not enough to detect a ten-point difference. Detecting an effect that small at 80% power needs roughly 218 paired problems, and running those costs a day of box time during which the endpoint serves nobody. So our result does not refute the leaderboard; it means that on the one coding benchmark we run ourselves, with both models measured the same week on the same hardware, we could not find the gap.

We also measured coding and long-context retrieval, and nothing else. General reasoning, instruction following and multi-turn agentic behaviour are untested here, and the published differences on those may well be real. What we can say is that on an automated agentic platform running long autonomous builds, several basic coding tasks that took an hour on the dense model finish in about ten minutes on this one, with no quality drop we could see in the output.

The honest shape of the decision is this: we traded a difference we could not measure for a difference we could not miss. A request that queues, or streams at half speed, is a worse outcome for the person waiting than a marginally weaker answer that arrives promptly - and a model too slow to serve the demand is worth nothing at all to the requests it turns away.

What is running now

Qwen3.6 35B-A3B in NVFP4, on the same GB10 behind the same TLS proxy, at the same price: $0.20 per million input tokens and $0.80 per million output. It answers to qwen3.6-35b-a3b, keeps the full 262,144-token context, reads images, calls tools and returns its reasoning. Qwen3.8 27B is retired from the catalog, and because it was served only from our own hardware it is simply gone rather than routed elsewhere.

Its configuration file stays in the repository, and switching back is one line. Two things are still open, and we had both of them wrong when this post first went up. The draft model does not need publishing by anyone: this checkpoint already carries a multi-token-prediction head, one attention-and-experts layer trained to guess the token after next, sitting unused in weights we had already downloaded. And the kernel is not a matter of asking for CUTLASS, which no W4A16 checkpoint can use; it means either a build of this model quantized for FP4 activations, or vLLM's fused SM12x kernel, which is written for exactly this chip and simply is not picked automatically. Both are measurements we owe you rather than improvements we can claim, and every number above was measured without either.