MTP vs DFlash2 on CARouter's DGX Spark: 35% more tokens per step, not a faster step

CARouter Cloud is the part of CARouter we host ourselves: one ASUS Ascent GX10 (an NVIDIA GB10 "DGX Spark"-class box, 128 GB of unified memory, arm64) running vLLM behind a TLS proxy, serving two embedding models and, since the previous article, Qwen3.8-27B as the coding model. That article ended on a memory finding. This one is about speed, and it starts with a coincidence: on 2026-08-26 a ByteByteGo newsletter titled "How to Make LLMs 3X Faster", a clean explainer of speculative decoding, landed in the inbox, and the same morning vLLM 0.28.0 was published with the one drafter our model could not use yet. So we checked the article against the box line by line, then ran the experiment it had just made possible. Here is what got better, what did not, and what a "3×" headline is worth on unified memory.

The previous article: Laguna S 2.1 vs Qwen3.8-27B on CARouter's DGX Spark, speed lost to memory.

The short version

• Already there. Qwen3.8-27B was already running the article's technique: its built-in multi-token-prediction (MTP) head, re-run three times, was landing 2.76 tokens per step on production traffic - 75%, 57% and 44% at the three positions, read from 22 hours of the gateway's own requests.
• The lever. The one thing left was a better drafter. DFlash2, a five-layer block-diffusion draft for Qwen3.8-27B, needs vLLM 0.28.0; we bumped the image and measured MTP against DFlash2 at 1, 2, 4, 8 and 16 concurrent streams, at three draft lengths.
• The result. With a five-token block, each target step yields 3.45 accepted tokens against 2.55 (+35%), so decode is 28.5 tokens/s single-stream against 20.8 (+37%), 91 against 81 at four streams, 216 against 203 at sixteen. Where the gateway actually runs, one or two coding agents at a time, that is a third more per stream.
• The estimate we got wrong. We predicted 1.6-2.4× because the new draft reads 3.85 GB per pass. The step did not get faster at all: 121 ms before, 121 ms after. Every token gained came from more tokens accepted per step - 3.45 against 2.55, +35% - not from a faster step.
• The cost. 3 GiB of weights, a fifth of the KV budget (3.2 concurrent 262K contexts instead of 3.9), a seven-minute cold start, and vLLM's newer V2 model runner.
• Shipped. Qwen3.8-27B NVFP4 with a five-token DFlash2 draft on vLLM 0.28.0, all four containers on the new image. The MTP fallback is one line in one file.

What the article says, checked against the box

The article's argument is the standard one, and it is correct: generating a token means reading every weight once for a handful of arithmetic, so decode is bound by memory bandwidth and the math units idle; a small drafter can guess several tokens ahead, and the big model can check all of them in a single pass for about the price of one, keeping the guesses that match. The gain is the acceptance length - how many draft tokens survive per step - and it depends on the workload (code and extraction accept well, open-ended prose does not), on the sampling temperature (higher flattens the distribution and lowers acceptance), and on load (under a large batch the spare compute the trick spends is no longer spare). The article lists four places a draft can come from: a separate small model, prediction heads trained into the big one, a cheaper copy of the same model, and a search over the text already seen.

We went through that list against what was running. The prediction heads: that is Qwen's MTP head, already on, and vLLM's counters said what it was worth on real traffic - 208,647 tokens drafted, 122,270 accepted, 2.76 tokens per target step. The search over existing text (n-gram or suffix drafting): vLLM runs one proposer per engine, so it would replace the MTP head rather than add to it, and at 2.76 tokens a step the head is hard to beat on code the model has to write, as opposed to code it copies. A separate small model: a worse drafter than a head that reads the big model's own hidden states, and it costs memory. The cheaper-copy idea (QuantSpec): not in vLLM. The load cliff: the article says vLLM can disable speculation above a batch size, and it can - the modern form is a per-batch-size schedule of draft lengths - but on this box a step at our ceiling of 16 sequences carries at most 64 tokens, and a 27B on a GB10 is nowhere near compute-bound below a thousand, so we had no cliff to protect against.

What the check did surface is the shape of our traffic: 41 million prompt tokens for 192 thousand generated, 96% of the prompt tokens served from the prefix cache, which means a typical request spends about two seconds on the four thousand tokens it has to prefill and twenty-five on the 460 it generates. Decode is nine tenths of the wall clock. Speculation is exactly the right lever, and the only thing left to pull on it was the drafter.

The one thing left: a better drafter

DFlash2 (z-lab, ICML 2026) is a drafter trained for Qwen3.8-27B: five small transformer layers that read the big model's hidden states at five depths and, instead of guessing one token at a time, predict a whole block in one pass and let a small selector trace a path through the candidates. Its authors report 5.5 accepted tokens per step on GSM8K against 5.0 for the MTP head at the same draft length, and 3.4× over plain decoding. It ships as a 3.85 GB checkpoint and needs an architecture vLLM added in 0.28.0 - released, as it happened, that morning.

Before running it we wrote the estimate down, as we did last time. Our MTP step was ~123 ms, of which the three sequential re-runs of the single MTP layer were, by the earlier sweep's arithmetic, about 20 ms each. DFlash2 replaces them with one pass over 3.85 GB, which at this box's 273 GB/s is about 15 ms, and verifying eight positions instead of four is nearly free on a bandwidth-bound step. So: a ~100 ms step, and if acceptance went from 2.6 to somewhere between 3.5 and 5, single-stream decode would go from 18-20 tokens/s to 30-45. That was the prediction. Keep it in mind.

What we measured, and how

Same probe as the previous article, the bench script in our infrastructure repository: N concurrent code-generation requests through the production proxy, decode rate read from vLLM's own counters over a 40-second window, then a fresh 12K-token prefill. Two additions this time. The probe now takes the sampling temperature as a parameter, because the article's temperature point deserved a number: 0.7 is what every sweep so far used, 1.0 is the checkpoint's default and what the gateway sends when a client sets nothing. And single-stream points were repeated, because the first DFlash2 numbers spread more than we expected, and one 40-second window of one generation is not a measurement of a drafter whose acceptance depends on what the model happens to be writing.

Every configuration is one reboot of the same served checkpoint (unsloth's NVFP4 quantization of Qwen3.8-27B, 262K context, 36 GiB of KV) on the same box, the same day, one after the other.

Finding 1: the image bump alone changed nothing, and sixteen streams still scale

vLLM 0.28.0 ships the same CUDA 13.0.2, torch 2.13 and FlashInfer 0.6.16 stack as 0.27.1, picks the same CUTLASS NVFP4 kernel and the same FlashInfer attention on this silicon, and with the MTP head it decodes 20.8, 39.7, 81.2 and 124.4 tokens/s aggregate at 1, 2, 4 and 8 streams where 0.27.1 did 19.8, 40.0, 74.9 and 127.4 - noise. Prefill was 2,007 tokens/s against 2,083, one probe each. The one new cell is 16 streams: 202.7 tokens/s aggregate, 1.63× the eight-stream figure. The cliff the article warns about, where speculation starts costing more than it returns, is not there at our maximum batch.

Finding 2: 35% more tokens per step; the step did not get faster

DFlash2 at its native block of seven draft tokens: 26.0, 52.3, 81.0, 135.2 and 192.6 tokens/s aggregate at 1, 2, 4, 8 and 16 streams, against MTP's 20.8, 39.7, 81.2, 124.4 and 202.7. Up 25% and 32% at one and two streams, level at four, up 9% at eight, and down 5% at sixteen.

Then the number that explains it. At one stream DFlash2 accepted 3.16 tokens per step and decoded 26.0 per second: a 122 ms step. MTP accepted 2.55 and decoded 20.8: a 123 ms step. The step is the same. One pass of a five-layer drafter plus its selector plus an eight-position verify costs, on this box, exactly what three re-runs of a single MTP layer cost. Our estimate had priced the draft at its bandwidth, 3.85 GB in 15 ms, and forgot that the step also runs Qwen's Gated DeltaNet layers over eight query positions instead of four, the selector, and a verify that is only free while the batch is one. Everything DFlash2 gained, it gained by accepting more tokens per step; it gained nothing by being faster per step.

And the acceptance tells you where the block should end. On the probe DFlash2 lands 75%, 52% and 36% of its first three tokens, about what the MTP head lands on production traffic, then 28%, 22%, 15% and 12% for positions four to seven. Those four positions add 0.77 accepted tokens per step. At one or two streams they are free, because the step is bandwidth-bound and verifying four more positions costs no time; at sixteen streams a verify of eight positions across sixteen sequences is 128 tokens of real work per step, and it is what turned +9% at eight streams into −5% at sixteen.

Finding 3: seven is too deep, five is the block

So we measured shorter blocks. At five draft tokens: 28.5, 47.5, 91.4, 137.4 and 216.1 tokens/s across the five concurrencies, the one- and two-stream points averaged over two runs each. At four: 23.0, 47.8, 94.2, 142.4 and 229.7. The three DFlash2 blocks and the MTP head all run the same ~121 ms step; what differs is the yield - 2.55 tokens for MTP, 2.77 at four, 3.45 at five, 3.16 at seven, single-stream - and how much the verify costs once the batch grows. Five is the block that wins where the gateway lives, one and two coding agents at a time: +37% over MTP at one stream. And it keeps the gain at every batch size we can reach, +13% at four, +10% at eight, +7% at sixteen. Four edges it by 3-6% from four streams up and loses a fifth at one; seven wins nothing that five does not.

The article's "shrink the draft as concurrency rises" knob exists in vLLM as a per-batch-size schedule of draft lengths, and the scheduler honours it; but the DFlash2 speculator in 0.28.0 drafts a fixed block, so we did not try it blind on a production engine. A static five does the job on this box.

Finding 4: temperature is a real lever, and it lives on the client

The article says a higher sampling temperature lowers acceptance. It does, and it is not small. At the checkpoint's default temperature of 1.0 the MTP head accepts 43% instead of 52% and single-stream decode falls from 20.8 to 18.5 tokens/s: 11% for a parameter most clients never set. DFlash2 at that temperature is where the spread showed up. Four single-stream runs at the seven-token block gave 32.5, 26.8, 21.3 and 17.1 tokens/s, acceptance from 43% down to 16% on the same prompt family, a mean of 24.4 and a worst run below MTP's 18.5; the five-token block gave 19.8 and 22.1, the four-token block 22.3 and 20.3. MTP, whose head is steadier, spreads 18 to 21 across three days of measurements. A block drafter's acceptance depends on what the model happens to be writing in the window, which is why every DFlash2 number in this article is a mean with its spread and why the probe now repeats.

The practical reading is on the client side: a coding agent that sends temperature 0.7 for code instead of leaving the default gets a tenth of its decode back before any server change, and the gateway is one place that default could live.

What that means for our clients

Reduced to what a person using the coding endpoint sees:

• Answers are written about a third faster. Single-stream, tokens now arrive every 35 ms instead of 48. The average response on our traffic is about 460 tokens, so it completes in roughly 16 seconds instead of 22; a 2,000-token file rewrite takes about 70 seconds instead of 96.
• The first token arrives at the same moment as before. That wait is prompt processing, which speculative decoding does not touch: about two seconds per request on our traffic, because 96% of the prompt is already in the prefix cache from the previous turn.
• The gain shrinks when the box is busy, but it no longer disappears. With four to sixteen agents decoding at once, each gets 7 to 13% more than before instead of 37%. The native 7-token block lost ground at sixteen streams; the served one does not.
• The code is exactly the same. Speculative decoding only accepts tokens the model would have produced anyway, so quality, tool calls and image input are untouched, and the BigCodeBench pairing from the previous article still applies.
• Some responses feel faster than others. A drafter guesses well on predictable code (boilerplate, repeated patterns, edits that echo the file) and poorly on prose or unusual logic, so the speed-up varies with what is being written and averages out over a session.
• One thing is in the client's hands. An agent that sends a sampling temperature of 0.7 for code, instead of leaving the model's default of 1.0, gets roughly another 10%, because the draft is accepted more often when the model is less random.

What it costs

The draft is 2.95 GiB of weights (vLLM's loading line went from 21.97 to 24.92 GiB). Its own 2048-token sliding-window cache is carved from the same 36 GiB KV budget, which now holds 838,501 tokens instead of 1,014,892: 3.2 concurrent 262K-token requests instead of 3.9. Cold start is seven minutes instead of six. And 0.28.0 forces its V2 model runner for a DFlash2 draft, a newer code path, which served chat, separated the reasoning from the answer, parsed a tool call and read an image correctly in the same-day checks, and survived sixteen streams at seven draft tokens where the previous release had faulted at four streams with four MTP tokens.

What we deployed

Qwen3.8-27B NVFP4 with a five-token DFlash2 draft on vLLM 0.28.0; the two embedding servers on the same image; same proxy, same served name, same 262K context and 36 GiB of KV. The MTP head stays in the checkpoint and its configuration stays in the spec as one commented line, because a drafter whose acceptance swings between 16% and 43% on the same prompt deserves a fallback that swings less.

The honest caveats:

• DFlash2 was trained against Qwen's bf16 weights; we serve a community NVFP4 quantization, and the drafter reads hidden states the quantization perturbs. Some of the spread may be that. A draft trained on the quantized target is the experiment we have not run.
• Every figure is one box, one probe, one day. The one- and two-stream cells are means of two to four runs; the rest are single runs.
• The estimate was wrong by the whole cost of the draft pass. The lesson generalises: on unified memory, price a drafter by measuring its step, not by dividing its bytes by the bandwidth.
• Coding quality is untouched by construction - speculative decoding accepts exactly the tokens the big model would have produced - so the BigCodeBench pairing from the previous article stands.

What we got out of the day, beyond the number

A third more per stream is the headline, but the infrastructure came out of it with more than that: the image is on vLLM 0.28.0 with the upgrade procedure exercised end to end; the probe takes a temperature and repeats; the production counters now tell us the acceptance length of real traffic, which is the number to watch as the draft meets agents rather than a red-black-tree prompt; and forty-one evaluation sandboxes left over from the previous week's BigCodeBench run are gone from the box. The next lever is documented rather than guessed: block verification, a lossless acceptance rule the V2 runner exposes, and a drafter paired to the quantized weights.