Tuning speculative decoding on CARouter's DGX Spark: two thirds faster coding answers, same hardware

Speculative decoding has a small "draft" model guess several tokens ahead so the big one can check them all in a single pass, keeping the ones it agrees with. Which guesses the draft makes, and how they get checked, turn out to be two settings we had left at their defaults. Changing them - two keys in one config file, no new hardware, no change to what the model writes - took the coding endpoint on our self-hosted box from 20.8 to 34.8 tokens a second per stream: two thirds faster than the model's own built-in guesser, and 22% faster than what we shipped a week ago. Here is what each setting does, why each one is a loss on its own, and the conclusion from last week that the pair of them overturned.

The previous article: MTP vs DFlash2 on CARouter's DGX Spark, 35% more tokens per step, not a faster step.

The short version

• What a client sees. The coding endpoint writes an answer two thirds faster than it did before any of this tuning: a token every 29 ms instead of 48, a typical 460-token reply in about 13 seconds instead of 22. Nothing about the answers themselves changes.
• The setting that paid. draft_sample_method: "probabilistic" makes the draft sample from its own distribution at the request's temperature, shaken by the same random noise as the big model's own sampling, instead of always taking its single best guess. The acceptance test then compares two real distributions instead of reading a probability off a certainty.
• Neither setting pays alone. Block verification, the lossless acceptance rule we recommended at the end of the last article, loses on its own: 22% down at two streams and 11% down at eight, at the temperature clients get by default. Probabilistic drafting alone wins from two streams up and gives up a tenth of single-stream decode, which is exactly where a coding agent sits. Together they cover each other.
• And we had been wrong in an interesting way. Last week we measured the drafter's deepest guesses, found they did not pay for themselves, and shipped a shortened block. That was true of the default setting, which we had not noticed was a premise at all.
• So the depth answer flipped. With the draft sampled probabilistically we re-ran the block-length sweep over four, five, six and seven. The drafter's native seven-token block won: 34.8 tokens/s single-stream against the five-token block's 29.3, and 28.6 against 24.7 at the temperature clients get by default.
• The cost. Nothing measurable. The draft's cached distribution is about 106 MiB, absorbed; the weight load is unchanged at 24.92 GiB; the KV budget gives up 3% to the deeper block's own cache; prompt processing does not move.

What we said last week, and why it was conditional

Last week's sweep found that positions four to seven of the drafter's native block were accepted 28%, 22%, 15% and 12% of the time. Those four positions added 0.77 accepted tokens per step, which is free at one or two streams and real work at sixteen, so shortening the block to five kept the single-stream gain and turned the batched regime positive at every size. We wrote "a static five does the job on this box" and shipped it.

Read that again with a week's hindsight. Every number in it is an acceptance number, and acceptance is not a property of the box, or of the model, or of the workload alone. It is a property of how the draft is produced and how it is checked. We measured one drafting rule - take the drafter's best guess, every time - and drew a conclusion about block depth as though depth and drafting were independent. They are not, and this is the part of the whole exercise that generalises: whenever the drafting rule changes, the depth question has to be asked again.

The setting: sampling the draft the way the model samples

vLLM's speculative configuration has a key we had dismissed. draft_sample_method defaults to "greedy": the drafter proposes its single most likely continuation, and the acceptance test treats that proposal as a certainty - probability one on the chosen token, zero everywhere else. Set it to "probabilistic" and two things change.

The first is in the drafter. DFlash2's selector walks a beam of sixteen candidates per position; in the default mode it takes the highest-scoring one, and in probabilistic mode it samples among them at the request's own temperature. The kernel that does this carries a comment worth quoting, because it is the whole idea in one line: "Candidate ids key the noise, matching the target's own sampling." The draft and the big model are shaken by the same random noise, so when the big model samples stochastically the draft tends to make the same stochastic choice.

The second is in the checker. With a real distribution behind it, the drafter's scores are cached and the acceptance test becomes the true ratio of the big model's probability to the draft's, instead of a probability read off a certainty. That is what makes the second setting - block verification - mean anything at all.

We had written in our own notes that this option applied to Qwen's built-in head and not to this drafter. That was wrong, and reading the code is what corrected it: the DFlash2 speculator overrides the exact method that allocates the cache this mode needs, specifically so that it works with it.

Finding 1: the lossless setting we recommended, alone, is a loss

Last week's closing line named block verification as the next lever: a lossless acceptance rule that checks the whole drafted block jointly rather than one token at a time. It is genuinely there, it works with this drafter, and it costs no memory - the boot prints the same weight load and the same cache size. It also loses. At temperature 0.7 it lands inside the existing spread everywhere, which is what the theory predicts: block verification is provably identical to the ordinary rule when nothing is sampled at random, and a code probe at 0.7 is close to that. At temperature 1.0, where the gain was supposed to be, it gives 37.8 tokens/s at two streams against 48.7 and 120.1 at eight against 134.7 - and that eight-stream figure is three runs within 2.2 tokens/s of each other, so it is not noise. Acceptance never rose.

The reason is the one the previous section sets up. A best-guess draft is a certainty, so the joint rule has almost nothing to work with that the token-by-token rule was not already getting, while its extra arithmetic is charged on every step and bites once the batch is wide. A lossless rule is not automatically a free one.

Finding 2: probabilistic drafting wins at batch and loses where the agent sits

Sampled probabilistically at the five-token block, decode is 25.7, 57.1, 101.2, 163.5 and 246.0 tokens/s at one, two, four, eight and sixteen streams, against 28.5, 47.5, 91.4, 137.4 and 216.1 for the same block with the default setting. From two streams up that is +20%, +11%, +19% and +14%, and acceptance rises with it - 3.0 to 3.9 tokens per step where the best-guess draft managed 2.7 to 3.0. At one stream it gives up a tenth, and one stream is where a coding agent actually sits.

At the temperature clients get by default the picture inverts: single-stream goes to 26.0 against 21.0, +24%, because coupling the draft's randomness to the big model's matters most exactly where the big model is most random. A setting worth +24% or -10% single-stream depending on a parameter the client controls is not a setting you ship on its own.

Finding 3: together, and then the depth answer flips

Adding block verification on top of probabilistic drafting repairs the single-stream cell - 29.3 tokens/s against 25.7 - because now there is a real draft distribution for the joint rule to work with. That combination has no measured cell below the ordinary rule at either temperature; its worst is -2%. So we re-ran the block-length sweep under it, overnight, four lengths at two temperatures with the noisy cells repeated three times.

Seven - the block the drafter was trained to predict, and the one we rejected a week ago - wins by 19% single-stream at temperature 0.7 and 16% at 1.0, and by 9% and 12% at two streams, for 2 to 5% given back at four and sixteen. Since the gateway runs one or two agents at a time, that is the trade we want, and it is the same trade we made last week when we chose five over four.

The convincing evidence is not the throughput, which is noisy on this box, but the acceptance: 3.85 to 4.83 tokens per step at seven against 3.18 to 3.92 at five, with six between the two and four below both. A clean ordering in the direction the mechanism predicts is worth more than any single fast run.

Finding 4: one run is not a measurement, and it nearly cost us

Probabilistic drafting alone produced 178.5 tokens/s at eight streams - its best cell, and 13% ahead of the combination we were about to ship. We flagged it in the merge request as a single run and therefore not to be trusted, and re-measured it overnight: 161.8, 167.8, 160.9, a mean of 163.5. The 178.5 was an outlier, the real gap is about 4%, and the decision it would have overturned was correct.

We are recording this because it is the second time in two articles that this box's variance has nearly written the wrong conclusion, and because the discipline that caught it is cheap: state the run count next to every number, and re-measure anything that would change a decision on its own.

What that means for our clients

• Answers are written two thirds faster than before any of this tuning. Single-stream at temperature 0.7, tokens arrive every 29 ms, against 35 ms after last week's change and 48 ms with the model's built-in guesser. The average response on our traffic is about 460 tokens: roughly 13 seconds, against 16 last week and 22 before.
• At the default temperature the gain is larger than last week's, not smaller. A client that sets nothing gets 28.6 tokens/s where last week's configuration gave 21.0. That is the case this change helps most, and it is the most common one.
• A busy box now shares the gain properly. With four to sixteen agents decoding at once each gets 19 to 31% more than the built-in head, where last week's block gave 7 to 13%.
• The first token still arrives at the same moment. Speculative decoding does not touch prompt processing, and it did not move: about 2,000 tokens/s throughout.
• Setting a temperature is still worth more than we can give you. A client that sends 0.7 for code instead of leaving the default 1.0 gets about 22% more decode, up from the 10% we quoted last week.
• The code is unchanged, by construction. Speculative decoding accepts exactly the tokens the big model would have produced anyway - though see the first caveat below, which is about how well we have checked that rather than whether it is true.

What it costs, and what we deployed

The draft's cached distribution is about 106 MiB, absorbed without changing anything else: the same 24.92 GiB weight load, the same seven-minute cold start. The deeper block carves more of the 36 GiB cache budget for its own use, so the budget holds 813,427 tokens instead of 838,501 - 3.10 concurrent 262K contexts instead of 3.20, a 3% trim. Prompt processing is unchanged.

Shipped: Qwen3.8-27B NVFP4 with a seven-token DFlash2 draft, draft_sample_method "probabilistic" and rejection_sample_method "block", on the same vLLM 0.28.0 image, same proxy, same served name, same 262K context. The two settings go together in the spec with a comment saying so, because either one alone is a regression somewhere.

The honest caveats

• We have not verified losslessness empirically. Speculative decoding accepts exactly the tokens the big model would have produced - that is a property of the construction, and it is why we keep saying quality is untouched. But we changed the sampling path and have confirmed only that chat, tool calls, image input and reasoning still work, not that the distribution of the output is unchanged. The benchmark run that would have checked it failed on a bug in our own evaluation harness, which had been collapsing per-task settings onto every task in a multi-task run; that is fixed now, and the check is still owed.
• The single-stream cells are wide. The three runs behind the served block's 34.8 were 39.7, 33.1 and 31.5. Every figure here is a mean with its run count, and the batched cells are two to three runs where last week's were often one.
• One box, one probe, one night. Same GB10, same checkpoint, same code-generation probe as both previous articles, each configuration a separate reboot.
• The drafter still does not match the weights it drafts for. DFlash2 was trained against Qwen's bf16 weights; we serve a community NVFP4 quantization, and the drafter reads internal states the quantization perturbs. Some of the single-stream spread is probably that. A draft trained on the quantized target remains the experiment we have not run, and it is now the only lever left that we know of.

What we learned that outlives the numbers

Two things, and neither is about this box.

The first is that a tuning result can carry a premise you did not write down. "Seven is too deep" was a fact about one drafting setting wearing the costume of a fact about hardware. It survived a week because nothing in how we recorded it made the premise visible, and the fix is not to measure more but to name the conditions a number depends on when you write it.

The second is that "lossless" and "free" are different words. Block verification changes nothing about what the model produces and still cost us 11% at eight streams, because a correctness guarantee says nothing about what the guarantee costs to compute. The estimate we got wrong last week priced a drafter by dividing its bytes by the bandwidth; the estimate we would have got wrong this week priced an acceptance rule by reading its proof.