Laguna S 2.1 vs Qwen3.8-27B on CARouter's DGX Spark: speed lost to memory

CARouter routes AI requests to Canadian-served providers, and CARouter Cloud is the part we host ourselves: one ASUS Ascent GX10 (an NVIDIA GB10 "DGX Spark"-class box, 128 GB of unified memory, arm64) running vLLM behind a TLS proxy, joined to our Kubernetes cluster as capacity. It serves two embedding models and one coding model. This is the story of choosing the coding model, told with the numbers, because the number that decided it was not the one we expected.

The short version

• Speed. On our code-generation probe Laguna decodes 46.5 tokens/s single-stream and 176 aggregate at eight streams; Qwen 18.3 and 122. Laguna leads at every concurrency we tried.
• Reasoning traces. On the text a coding agent actually produces, the two are level per stream: about 17 tokens/s against 14.
• Memory. Qwen at 44 GiB resident leaves 62 GiB free; Laguna at 98 GiB leaves 8. That is 3.9 concurrent 262K-token requests against 1.2, and it is the number that decided it.
• Quality. 33 against 31 on BigCodeBench's first 50 problems, McNemar p = 0.73. Not a gap.
• Shipped. Qwen3.8-27B in a community NVFP4 quantization, MTP depth 3, 36 GiB of KV. Laguna stays one environment flip away.

The two candidates

Poolside Laguna S 2.1 was the incumbent: a 117B mixture-of-experts with 8.5B active parameters, served from Poolside's NVFP4 checkpoint with their DFlash speculative-decoding draft. It is a very good coding model, and it is a memory hog: ~98 GiB resident at 262K context, which on a 121.6 GiB box leaves room for the two embedders and nothing else.

Qwen3.8-27B was the challenger: a dense 27B, 64 layers of which 48 are Gated DeltaNet (linear attention) and 16 are full attention, native 262K context, a vision tower, and a multi-token-prediction (MTP) head trained into the checkpoint. Qwen's own card puts it near the frontier on agentic coding (SWE-bench Pro 61.7, Terminal Bench 2.1 73.0).

The naive expectation, which we wrote down before measuring: the dense 27B would be much slower. A GB10 has about 273 GB/s of memory bandwidth, decode is bandwidth-bound, and a dense 27B at FP8 reads ~27 GB of weights per token where Laguna's MoE reads ~4 GB. Roofline arithmetic said 10 tokens/s against 60. On a plain code prompt the arithmetic held up: Laguna is two and a half times faster single-stream and ahead at every batch size we tried. What the arithmetic missed is the workload a coding agent actually sends, and the memory the smaller model leaves behind - which is where the rest of this piece goes, and where the decision came from.

What we measured, and how

Every throughput figure here comes from one probe, the bench script in our infrastructure repository: N concurrent code-generation requests through the production proxy, decode rate read from vLLM's own token counters over a 40-second window, then one fresh 12K-token prefill. Same probe, same box, same day for both models, so the columns line up. Where a figure comes from the eval run instead - the decode rate on long reasoning traces, read from the same counters while BigCodeBench was running - it says so, because the two workloads do not give the same answer. Coding quality is BigCodeBench, the first 50 problems, uncapped output tokens, scored in a sandbox - the same 50 problems for both, so the two runs pair.

Finding 1: speculative decoding is worth 5 tokens a step on easy code and 1.5 on reasoning

Laguna's DFlash draft proposes 7 tokens per step. On our code-generation probe it lands 58% of them - 5.1 tokens per step - and Laguna decodes 46.5 tokens/s single-stream. On the long reasoning traces a coding agent actually produces (its BigCodeBench run, read from the same counters), acceptance falls to 21%, 1.48 tokens per step, and the per-stream rate to about 17. Qwen's single MTP layer, re-run three times, accepts about half at each position on either workload: 2.5 tokens per step, 18.3 tokens/s single-stream on the probe and about 14 per stream with eight eval requests in flight. Laguna's speculation is spectacular when the text is predictable and ordinary when it is not; Qwen's is steady, and it still started out slower, because its base step reads 27 GB.

So the lever for Qwen was the checkpoint. We swapped Qwen's official FP8 weights for a community NVFP4 quantization (unsloth's: MLPs at 4-bit, attention projections at FP8, KV scales and the MTP head included). vLLM put it on FlashInfer's CUTLASS NVFP4 kernel - the same kernel family that makes Laguna fast on this silicon - and the step time fell from 185 ms to 130 ms, more than the byte ratio alone predicts.

Twelve configurations later, the shipped one is NVFP4 at MTP depth 3. Depth 4 tied depth 3 single-stream and then crashed vLLM with an illegal memory access at four concurrent sequences, which is the kind of thing you only learn by running it. Larger prefill chunks made prefill a third slower. KV-cache precision turned out to be a calibration decision, not a speed one.

Finding 2: on throughput, Laguna wins the easy case and ties the hard one

A coding endpoint behind a gateway is never one request at a time, so we measured both models at 1, 2, 4 and 8 concurrent streams, same probe, same day.

Laguna leads at every count: 46.5, 52, 96 and 176 tokens/s aggregate against Qwen's 18.3, 40, 73 and 122. Qwen's curve is the straighter one - within 5% of linear to eight streams, because a dense model pays its weight read once per step whatever the batch, and 32 tokens per step is nowhere near compute-bound for a 27B. Laguna's flattens between one and two streams, where its draft's acceptance dips, then climbs 3.8x by eight. The crossover we expected, where batching would erode the MoE's active-parameter advantage, is not there by eight streams on this box.

What narrows the gap is the workload. On reasoning traces, read from the same counters during the BigCodeBench runs, Laguna does about 17 tokens/s per stream at two streams and Qwen about 14 per stream at eight (113 aggregate); Laguna above two streams on that workload is the one cell we did not measure. Per stream, on the text agents actually produce, the two are level; on code Laguna can predict, it is far ahead. Prefill is Laguna's at every length: 3,550 tokens/s against 2,088 on a 12K prompt, 2,836 at 55K and 1,634 at 200K.

Finding 3: 62 GiB free is four 262K sessions instead of one

This is the number that decided it. Our requirement was concrete: 262K context, and several coding clients in flight at once. Context lives in the KV cache, and the KV cache lives in whatever memory the weights leave behind.

Laguna at 98 GiB resident leaves 8 GiB free after the embedders. Its 12 GiB KV budget holds 1.2 full-context requests, and there is nowhere to grow it. Qwen at 44 GiB leaves 62 GiB free. Its per-token KV is actually pricier than Laguna's (32 KiB against 24, because its 16 attention layers use 256-wide heads), but it can afford a budget three times the size, with room for four.

At the 36 GiB of KV we deployed, Qwen holds nearly four concurrent 262K-token requests (3.9: 1,014,892 cached tokens, measured); at 48 it would hold five. Laguna holds one, with a second waiting. For a gateway whose coding use cases are agents reading whole repositories, that is not a throughput detail; it is the product.

Finding 4: quality is a tie, which is what we needed

On BigCodeBench's first 50 problems - uncapped output, scored in the same sandbox, the same 50 for both - Qwen solves 33 and Laguna 31. Paired problem by problem the two agree on 42 of them: 28 both solved, 14 both failed. Qwen alone solves five, Laguna alone three, and an exact McNemar test on those eight discordant problems gives p = 0.73. That is not a ranking, and it was never going to be one at n = 50; it is the absence of a gap, which is the result the memory finding needed to mean anything. What did differ is how long Qwen thinks. It produced 462K output tokens for the set against Laguna's 264K, and ten of its samples ran past 16K tokens where Laguna never crossed 11K - reasoning traces of 20 to 31K tokens on the hardest problems, none of them hitting the 32K cap. Those samples are also how we found two ten-minute timeouts, one in the proxy and one in the eval harness, that had been silently cutting long generations off and retrying them; both are 7200 s now, and every number in this article was measured after the fix.

Long-context retrieval held too. On the same needle-in-a-haystack grid that validated Laguna's 256K (four context lengths from 32K to 220K, three needle positions), Qwen scores 9.0/10 overall - perfect at 95K, 9.0 at 157K, and 8.0 at 220K where Laguna held a perfect 10. The deepest end of the window is where the incumbent keeps a small edge; everything below it is a draw.

What we deployed

Qwen3.8-27B, NVFP4, MTP depth 3, 262K context, 36 GiB of KV, images on, behind the same proxy under the same served name. Laguna stays on disk and in the repository, one environment flip away, because a decision made on n=50 and one weekend should stay reversible.

The honest caveats:

• The NVFP4 build is a community quantization, not Qwen's.
• 50 problems is a signal, not a leaderboard.
• Laguna's decode on reasoning traces was measured at two streams only, so its eight-stream figure on that workload is an inference, not a measurement.
• The throughput comparison reversed on us late: until Laguna went through the same probe we were setting its eval-run rate against Qwen's probe rate and calling Qwen 3.6x faster at eight streams, which it is not. The decision stands on the other three findings.
• Everything here is one box, so the batch curves are a property of dense-versus-MoE on unified memory, not a universal law.

What a second DGX Spark would change

The obvious question after a weekend like this is whether a second GB10 - two DGX Sparks, linked over their ConnectX-7 200 Gb/s ports, the configuration NVIDIA sells for 405B-class models - would have changed the decision. It would have changed the options more than the answer.

• Both models, no choice. The simplest use of a second box is to serve Laguna on one and Qwen on the other, each with its embedders, and let the gateway route by model. Two coding endpoints, two sovereign offers, and Laguna gets its whole 121 GiB back for KV it never had. That is the configuration we would run if a customer asked for Laguna specifically.
• Qwen twice, KV everywhere. Two independent Qwen boxes double the concurrent 262K capacity to ten, add failover, and keep the linear batch scaling on each. This is the boring answer and probably the right one for a gateway: capacity is additive, nothing crosses the interconnect on the request path.
• Tensor-parallel Qwen across both. vLLM can split the dense 27B over two nodes, which halves the weight bytes each box reads per step and, in principle, nearly doubles single-stream decode. The price is an all-reduce per layer over a 25 GB/s link; at Qwen's hidden size that is a few kilobytes per token per layer, so the cost is latency, not bandwidth - plausible, unmeasured, and it spends the second box on single-stream latency, the one column where Laguna is already far ahead on code and the two are level on reasoning.
• A bigger model instead. The genuinely new option: with 256 GB the candidates change. Laguna's main revision (99.7 GB, the eight BF16 expert layers Poolside kept for quality) fits with room for context; a 122B-A10B MoE at NVFP4 (83.5 GB on the Hub) would have Laguna's active-parameter economy with a stronger base and still leave well over 100 GB for KV across the pair. That is the experiment we would run next, and it is a different article.

What a second box would not change is the shape of the finding. Decode on this silicon is bound by bytes per step and helped by tokens per step; a gateway is served by aggregate throughput and concurrent context; and a dense model that fits in a third of the memory turns the remaining two thirds into exactly the thing a gateway sells. More boxes multiply that. They do not replace it.