Qwen3.8-Flash-Next on CARouter's DGX Spark: better answers, for 10 times the wait
Qwen's Qwen4 architecture preview tops the coding leaderboards, and its smallest servable build is 125.9 GiB against a machine that has 121.6 GiB in total. 38% of it is a lookup table, so it runs anyway - streamed off the SSD, on a patched engine, alone on the box. It is about three times slower, holds a quarter of the conversations, and writes three and a half times as many tokens per problem. It probably codes better than what we serve, and on this hardware that advantage costs 13 minutes an answer.
Qwen released Qwen3.8-Flash-Next on 26 August, an early look at the architecture the Qwen4 family will be built on, and it went straight to the top of the coding leaderboards: 91.9 on LiveCodeBench v6 against the 80.4 of the model our own coding endpoint runs. Three days later we had it serving on the same desktop box that serves CARouter Cloud, and we did not keep it. Here is every number we measured, and what they cost.
The short version
- It does not fit. The smallest build we can serve is 125.9 GiB. The machine has 121.6 GiB of memory in total. This is the first model we have looked at that is larger than the computer, before a single token of conversation is stored.
- 38% of it is not a model. It is a lookup table: 20,000,000 rows of n-gram embeddings, 51.2B parameters, 47.7 GiB. A token touches 16 rows of it. Stream those from the SSD instead of keeping them in memory and what has to be resident drops to 78.2 GiB, which fits.
- The headline parameter count leaves it out. The model card says 125B. The weights hold 180B. The difference is that table, plus the draft head and the vision tower, all counted separately.
- It is about three times slower. 21.3 tokens a second on 1 stream against 62.2, and 129.8 against 367.2 with 16 streams in flight. Prefill is 3.7x slower. It holds a quarter of the concurrent 262K-token conversations: 1.7 against 7.0.
- It probably codes better. On the 40 problems it finished inside a 32,768-token budget it solved 29, against 27 for the model we serve and 26 for the one before it. It ran past the budget on the other 10. Neither margin is statistically significant.
- And it costs 10 times the wait. It writes 16,751 tokens per problem where the incumbent writes 4,738. At its own decode speed that is a 13-minute answer against 76 seconds, for at best 1 extra problem solved in 40.
A model larger than the computer
CARouter Cloud's self-hosted endpoint runs on one ASUS Ascent GX10, a DGX Spark-class desktop box: a GB10 chip with 128 GB of memory shared between the processor and the graphics side, of which 121.6 GiB is actually addressable. Every model we have shipped had to fit in that, next to the 2 embedding models and the reranker the product also serves.
Qwen3.8-Flash-Next does not. 9 different people had published quantized versions of it within 72 hours, and we checked all of them: the smallest one we could actually serve is 125.9 GiB of weights. That is 4.3 GiB more than the entire machine holds, before the operating system, before the inference engine, before storing one word of anyone's conversation.
Normally that is the end of the story, and our screening process is designed to end it there in about 10 minutes, from published metadata, without downloading anything. What made this one worth continuing was reading the actual list of tensors in the file rather than the model card.
The card says 125B parameters. The weights contain 180B. The gap is one enormous lookup table - the n-gram embeddings, exactly 20,000,000 rows of 2,560 numbers each, 51.2B parameters and 47.7 GiB even after every quantizer in the wild has had a go at it, and not one of them compressed it below 8 bits. It is 38% of the file. Add the draft head and the vision tower, also counted outside the headline, and 125B becomes 180B.
A lookup table is not a matrix multiplication. Producing a token touches 16 of those 20,000,000 rows, about 2.5 KB. So it does not need to be in memory at all: it can sit on the SSD and be read as needed. Somebody had already done exactly this - a public project that patches the official engine to memory-map the table from disk - and with it the resident weights fall to 78.2 GiB and the model runs on one box, alone.
What it took to run at all
6 patches on top of the official image, none of them ours: the memory-mapped table, two fixes for the linear-attention kernels on this particular chip, a guard on a state-copy race, a correction to prefix caching that was silently restoring an all-zero state on every cache hit, and a replacement for a sparse-attention kernel that returns different answers to the same question. The engine itself is not a release - it is a build from a branch, because the architecture is too new to be in any published version, including the one we pin.
It also cannot share the box. Our endpoint normally runs 4 services at once: the coding model, 2 embedding models and a reranker. Qwen3.8-Flash-Next takes the whole machine. Every measurement below was taken with the rest of the product's inference stopped.
The throughput
We ran the same benchmark we run on everything, unchanged: the same prompt, the same 20-second warm-up and 40-second measurement window, the same counters read from the engine itself.
Between 2.8 and 3.4 times slower at every concurrency, 3.7 times slower to read a prompt, and holding 1.7 full-length conversations at once where the model we serve holds 7.0. What makes this worse than it looks is the handicap: Qwen3.8-Flash-Next was running with speculative decoding - a draft head built into its own weights, accepting 51% of its guesses and delivering 2.02 tokens per step - on the fast kernel for this chip. The model that beat it has no draft model available at all and runs on a fallback kernel we have never managed to replace.
The coding score
50 coding problems, the same 50 we score every model on, run identically: same problems, same budget, same settings, one model after the other on the same box.
The first thing the run measures is not quality, it is length. Flash-Next is a reasoning model and it writes 16,751 output tokens per problem, against 4,738 for the model we serve. Our budget was 32,768 tokens, and it ran past that ceiling on 10 of the 50 problems - cut off mid-answer, scoring zero on each. Those are not wrong answers, so the comparable set is the 40 it finished inside the budget.
On those 40 it comes first: 29 solved, against 27 for the model we serve and 26 for the one before it. That is the same order as the public leaderboards, and neither margin is statistically significant - detecting the 11.5-point gap the leaderboards claim would take roughly 88 paired problems. Read it as consistent with Flash-Next being the stronger coder, and too small a sample to prove it.
The overrun rate is a result in its own right, and the one most likely to matter to you: 20% of ordinary coding problems need more than 32,768 tokens from this model. Whatever budget you set, it will hit it more often than the models you are used to, and a truncated answer is worse than a slow one.
What you actually wait for
The verbosity is not an artefact of our harness, though. It is the product.
3.5x the tokens, at a third of the speed, is 10x the wait. Seventy-six seconds becomes 13 minutes, for an answer that is at best 1 problem in 40 better and is usually identical. And a model that overruns a 32,768-token budget on a fifth of ordinary coding problems will do that to a paying caller too, not only to our test harness - a truncated answer is worse than a slow one.
The honest caveat
40 problems cannot settle whether one model is 10% better than another at writing code, and we are not claiming Qwen3.8-Flash-Next is worse. We think it is probably better. What we measured is that on this hardware its advantage is unreachable: it arrives ten times later, in a quarter of the parallel conversations, on an engine built from a branch, with nothing else running on the machine.
A different box changes that answer entirely. 2 of these machines, or one with more memory, and the table stays in memory, the model shares the hardware with nothing, and the comparison is a fair fight it might well win. 3 of the models we have rejected this month were rejected for the same reason, which is not really a fact about the models.
What would change our mind
- A quantized table. Nobody has compressed the n-gram embeddings below 8 bits, including the people who quantized everything else to 4. At 4 bits the file drops by 23.8 GiB and stays in memory.
- A quantized linear-attention path. 43% of what the model reads per token is projections that every published version leaves at full width. Halving those roughly doubles the ceiling.
- The architecture in a released engine. Today it exists on a branch and needs a kernel fix for this chip that upstream has not taken. That turns a patched fork back into a version number.
- A second box. The configuration everyone running this model successfully is using.
Until one of those, the coding endpoint keeps running a smaller, older, lower-scoring model that answers in 76 seconds and holds 7 conversations at once. Everything above is reproducible from our infrastructure repository: the screening with its arithmetic, the benchmark script, and the eval configuration.