Liquid AI released LFM2.5 24B-A2B on 13 October. It holds 24 billion parameters and reads about 2.4 billion of them for each token it writes.

Liquid compares it with Qwen3.6 35B-A3B, the model behind CARouter Cloud's coding endpoint, and reports that it is faster and better at agent work. We measured both on GB10s, the chip CARouter Cloud runs on, at 4 bits as we serve our own models, to check both claims.

The last model we tested this way: Kolibri-1 vs Qwen3.6 35B-A3B.

The short version

• Speed, on Liquid's engine. On llama.cpp, where Liquid measured its own figures, LFM2.5 writes 105 tokens a second for one request and Qwen 65. Liquid's claim holds.
• Speed, on ours. On vLLM, the engine CARouter Cloud serves with, Qwen stays ahead: 118 against 96 for one request, 486 against 387 with sixteen at once.
• Answers. LFM2.5 writes about 40% fewer tokens for the same answers. A typical coding answer arrives in 33 seconds against 40 for one user alone.
• Quality. Level on coding, 30 of 50 problems each. Better at following instructions, 95% against 89%. Worse at calling tools, 70% against 81%, and at science questions, 73% against 83%.
• Memory. LFM2.5 at 4 bits holds 12.9 GiB of weights, against 22.0 GiB for Qwen.
• What we decided. CARouter Cloud keeps Qwen3.6 35B-A3B. LFM2.5 reads half as much context, takes no images, and calls several tools at once less reliably.

Two engines, two different races

llama.cpp is built to run a model on one computer for one person. vLLM is built to serve many people at once from one machine, and it is what CARouter Cloud runs.

Qwen3.6 also ships with a small helper that drafts the next few tokens for the main model to check, which makes it about half as fast again on vLLM. Liquid has announced a helper of its own for LFM2.5, for a different engine, and has not released it yet.

Which LFM2.5 we ran

Liquid publishes a 4-bit build of LFM2.5 for llama.cpp and an 8-bit one, in FP8, for vLLM. For vLLM at 4 bits, the format we serve Qwen in, we took two builds made by the community.

They taught us something about this model. Two fifths of the parameters LFM2.5 uses for each token sit outside its experts, the specialist networks that 4-bit builds usually compress. One build compresses only the experts and writes 58 tokens a second. The other compresses nearly everything and writes 96.

Same model, same engine, 1.65 times the speed. Compressing the rest cost nothing we could measure on code or instructions, and some accuracy on tool calls: 349 of 500 against 371. Below, LFM2.5 on vLLM is the faster build, the one we would serve.

Memory

At 4 bits LFM2.5's weights are 9 GiB lighter than Qwen's. With the same 28 GiB set aside for conversations, it holds 41.7 GiB in all against 52.6 GiB.

Per token of conversation the two cost about the same. LFM2.5 fits more conversations in that space only because each one can be half as long: 131 072 tokens against 262 144.

Speed

On llama.cpp, LFM2.5 is 1.6 times as fast as Qwen and reads a prompt 1.74 times as fast. Liquid reports 114 against 67 tokens a second on an AMD Ryzen AI Max+ 395, a machine with about the memory speed of the GB10. We found the same gap.

On vLLM the order flips. Qwen's helper puts it 20% to 28% ahead at every load. Without its helper, Qwen writes 76 tokens a second alone on this same machine, less than LFM2.5's 96.

llama.cpp does not share a machine well. With sixteen requests at once, LFM2.5 totals 250 tokens a second on it, against 387 on vLLM.

LFM2.5 reads prompts faster on vLLM too: about 9 900 tokens a second against 6 500 for Qwen. Liquid's FP8 build was the slowest of all on this chip, at 54 tokens a second alone.

Tokens a second are not the whole story, because LFM2.5 writes less. For the same 50 coding problems it wrote 173 thousand tokens and Qwen 291 thousand. A typical answer took 33 seconds for one user alone on vLLM, against 40 for Qwen.

Quality

We use the same 50 BigCodeBench problems for every model we consider: the model writes a Python function, and the function is run against tests. Both solved 30, and 26 of them were the same problems.

IFEval gives 541 prompts with instructions a program can check, such as a word count or a required format. LFM2.5 followed every instruction of 95% of them, Qwen of 89%. The gap is too large to be chance, and it is the one Liquid reports.

Calling tools goes the other way. On 500 requests from the Berkeley function-calling benchmark, LFM2.5 got 70% right and Qwen 81%. With one call per request LFM2.5 is as good. When a request needs several calls at once, it usually makes one, and it calls a tool more often when none fits.

On 198 graduate-level science questions (GPQA Diamond), LFM2.5 answered 73% correctly and Qwen 83%. Liquid's own table shows the same gap, and Liquid recommends pairing the model with web search for questions of knowledge. It is also the one test where LFM2.5 wrote more than Qwen.

We also hid one sentence, naming the first band to play on the Moon, in documents of 16 000 to 100 000 tokens, and asked for it back. Qwen found it 12 times out of 12, LFM2.5 9. Once LFM2.5 answered The Beatles. Its window also stops at 131 072 tokens, against 262 144 for Qwen, so it cannot be asked about longer documents at all.

What we cannot tell you

• How LFM2.5 does inside an agent. Liquid trained it in OpenClaw, Hermes Agent, Pi and OpenCode, and we ran none of them.
• How fast it gets with its helper. Liquid has not released it.
• How it does in the fifteen other languages it was trained for. We tested English only.
• Whether a gap of a few points on our smaller tests is real. Fifty coding problems cannot rank two models this close.

What this means for CARouter

CARouter Cloud's GB10 keeps Qwen3.6 35B-A3B. On the engine that serves our customers, LFM2.5 writes more slowly but writes less, codes as well and follows instructions better. It also reads half as much context, takes no images, calls several tools at once less reliably, and is licensed for commercial use only to companies under USD 10 million of annual revenue.

Liquid built it for one person on one machine, and there it does what Liquid says: 105 tokens a second on a GB10 with llama.cpp, 1.6 times Qwen.

CARouter Cloud serves Qwen3.6 35B-A3B on two Intel Arc Pro B60 graphics cards in Ottawa, beside its GB10. The same cards now write up to 79% more tokens a second for each request.

The hardware, the model and its 4-bit form did not change. The time was being lost on the server's processor, not on the cards.

The article that introduced this server: Qwen3.6 35B-A3B on two Intel Arc Pro B60 cards.

The short version

• What a client sees. With two requests on a card, each one gets 64 tokens a second instead of 36. With four, 55 instead of 35. With eight, 44 instead of 35. Alone, 72 instead of 60.
• What we changed. The engine now records the model's computation once and replays it. Before, it ran every step through Python.
• What stood in the way. Two logging calls in Intel's build of vLLM. We remove them when we build our own image.
• How we tested it. 450 prompts in both modes, a tool call, a reasoning question, and prompts of 29 000 and 101 000 tokens.
• What it costs. A longer first start, and about 6% less room for the prompts the engine keeps in memory.

Where the time of a request goes

Our engines count what they do. Writing the answer is about nine tenths of the time a request takes, and a card usually writes for about two requests at once.

That is where this server was weakest. One request alone got 60 tokens a second. From two to eight requests on a card, each got about 35.

The card was waiting for Python

Our first article on this server noted that one processor thread paces each engine. This time we looked at what that thread does.

About 70% of each step went to the Python interpreter and to PyTorch's own bookkeeping, on that one thread. About a quarter went to handing work to the card. The card itself had time to spare.

That explains the flat 35. Most of a step's cost was fixed, whatever the number of requests.

Record once, then replay

vLLM can record the model's computation once, as a graph, and replay it at every step. The replay skips most of the Python work.

Intel's build of vLLM ships with that mode off, and for this model it did not start. The cause was small: two logging calls sit in code that PyTorch's compiler has to follow, and the compiler stops on them.

We remove those two calls when we build our own image. The problem, as reported to Intel.

What we measured

Same card, same day, in both modes, while the other card served customers.

With two requests on a card, each one gets 64 tokens a second instead of 36, which is 79% more. With four it is 55 instead of 35, and with eight 44 instead of 35. One request alone gets 72 instead of 60.

In total, one card writes 129 tokens a second with two requests instead of 72, and 349 with eight instead of 282. Reading a prompt is as fast as before, about 5 900 tokens a second.

The model's built-in helper, which did not pay on this server, now does for one or two requests: 111 and 88 tokens a second each. From four requests it is slower than without it, so it stays off.

What we checked before serving it

Intel describes this mode as not yet supported in that build. So we did not take the speed on trust.

We sent 450 different prompts to both modes and counted the answers that contain a long run of one repeated character, a known sign of trouble. There were 3 in each mode, all of them lines drawn in a table.

A tool call, a reasoning question, and a fact hidden in prompts of 29 000 and 101 000 tokens came back right in both modes. A sample of this size would show a frequent fault, not a rare one.

We then switched one card at a time, with its traffic moved to the other card first.

What it costs

The engine takes about twice as long to start the first time, because it prepares the graphs, and it keeps them for the next start.

It also leaves about 6% less room for the prompts it keeps in memory.

On a small card, the first thing to profile is the processor beside it.

CARouter Cloud serves Qwen3.6 35B-A3B on its own hardware in Canada. We serve it at 4 bits, a compressed form of about 20 GiB that fits a 24 GB graphics card.

At full precision, the same model is 67 GiB. AirLLM is an open-source library that says it runs such a model on a small card without compressing it.

We tested that on this model. Does it run within 24 GB, with images, video and the model's whole window, and how fast?

The previous article: this model served at 4 bits on two Intel Arc Pro B60 cards.

The short version

• The result. AirLLM runs the full-precision model in under 3 GiB of a 24 GB card. Text, images and video work, and the answers are the ones our serving engine writes.
• The price. One token every 10 seconds on a GB10 limited to 24 GB, and one every 68 seconds on an Intel Arc Pro B60. A 500-token answer takes 1 hour 25 minutes, or 9 hours 27 minutes.
• Against serving. Served at 4 bits, the model writes 74.9 tokens a second on our GB10 and 59 to 62 on an Arc Pro B60. AirLLM is about 760 and 4 100 times slower.
• The window. As it ships, AirLLM reads a prompt of about 78 000 tokens within 24 GB, of the 262 144 the model accepts. With a short loop of our own around it, a prompt that nearly fills the window is read, in 15 minutes on the GB10 and in 47 on the Arc Pro B60.
• Our reading. It is a way to get one reference answer from the full model on a small card. It is not a way to serve.

How a 67 GiB model goes through a small card

A model is a stack of layers, 40 for this one, and each token is computed by going through them in order. A serving engine such as vLLM keeps every layer on the graphics card, which is why the card must be as large as the model.

AirLLM keeps the model on disk, one file per layer. For each token it copies a layer to the card, runs it, frees it, and takes the next one.

The card never holds more than one layer and the work in progress. In exchange, every token moves the whole model, 65 GiB, through the card.

What we ran

AirLLM 4.0.0 on Qwen's own full-precision files of Qwen3.6 35B-A3B, one request at a time, on two machines.

The first is a rented GB10, the chip of our own machine, with 128 GB of memory shared between the processor and the graphics. We limited the software to 24 GiB of graphics memory, so it behaves like a 24 GB card in a machine with plenty of memory.

The second is a real 24 GB card: one Intel Arc Pro B60 in the server of the previous article, which has 32 GB of memory. Its other card kept serving customers during the test.

AirLLM's one-line example did not load this model, which is not on its list. Its own class for a sibling model, Qwen3.8 27B, named by hand, did. We changed nothing in AirLLM.

It fits, and it answers like our engine

A short prompt, an image and a short video clip each peaked under 3 GiB of graphics memory. The Arc Pro B60's driver counted 3.5 to 4 GiB in use, of 24.

We sent the same prompts to vLLM holding the same full-precision files. The 24 tokens of text, the sentence about the image and the answer about the clip were identical, to the character.

One token every 10 seconds, or every 68

On the GB10 limited to 24 GB, AirLLM writes one token every 10 seconds. A 500-token answer, about a page, takes 1 hour 25 minutes. Our GB10 serves the 4-bit model at 74.9 tokens a second: the same answer in 7 seconds.

On the Arc Pro B60, one token takes 68 seconds and the answer 9 hours 27 minutes. The same card serves the 4-bit model at 59 to 62 tokens a second.

Full precision is not what costs the most. vLLM with the same full-precision files, on a GB10 using all its memory, writes 29.6 tokens a second: the answer in 17 seconds.

Almost all the time goes to moving the model. Of the GB10's 10 seconds, 5.7 bring the next layer onto the card, 4.1 free the previous one, and 0.3 is computing.

The two machines differ by where the files are. The GB10's memory holds all 65 GiB of them, so a layer is copied from memory. The server with the B60 has 32 GB, so every token reads the whole model from disk again, at about 1.1 GB a second.

The window: 78 000 tokens as shipped, all of it in chunks

The model accepts up to 262 144 tokens at once: its window. AirLLM gives the model the whole prompt in one pass, and the memory used grows with the prompt, by about 0.2 GiB per thousand tokens.

Within 24 GiB, a prompt of 77 923 tokens passed and one of 93 938 did not. An optional component that AirLLM's own code mentions, flash-linear-attention, moved the limit: 126 007 tokens passed and 198 087 did not.

On the Arc Pro B60, a prompt of 30 094 tokens passed in one pass. One of 61 908 tokens filled the card, and we stopped that run.

We then wrote a short loop around AirLLM. It gives the model the prompt 8 192 tokens at a time and keeps, between two passes, what the model retains of what it has read.

With it, a prompt of 260 199 tokens, nearly the whole window, was read in 15 minutes within 19.2 GiB on the GB10, and a fact placed at its start was found.

On the Arc Pro B60, the same prompt was read in 47 minutes, with 16.3 GiB of the card in use at the peak, and the fact was found too.

Each chunk is a full pass over the model. vLLM reads the same prompt in about a minute.

AirLLM's own 4-bit mode

AirLLM has a compression mode of its own, to 4 bits, announced as up to three times faster.

As shipped, it did not start on this model. Its check of free disk space asked for 238 GB for a 67 GiB model, and the option that skips the check stopped on its first file.

With that check bypassed by hand, it ran on the GB10 at one token every 129 seconds: 13 times slower than without compression.

We see only one use case for AirLLM: checking what a full-precision model answers to a given prompt, on a machine that cannot hold it.

CARouter Cloud serves open models on its own hardware in Canada. Its first machine is built on an NVIDIA GB10, with 128 GB of unified memory.

We have been testing a second kind of machine in Ottawa: a server with two Intel Arc Pro B60 graphics cards, 24 GB each. We ran on it the model the GB10 serves, Qwen3.6 35B-A3B, at 4 bits.

The previous article: this model and its built-in helper on the GB10.

The short version

• The result. With 16 requests on each card, the two cards write 818 tokens a second in total.
• At equal load. With 16 requests in all, they write 558 tokens a second, about 35 each. The GB10 writes 342, or 479 with the model's built-in helper.
• One request alone. The GB10 stays ahead, at 74.9 tokens a second, or 114.9 with the helper, against 59 to 62 on a B60.
• Two cards as one pool. The Linux kernel refuses it on this generation of processor. With that one check lifted for a test, the pool worked, ran slower than one card, and needed the model's size again in the server's own memory.
• The processor. One processor thread paces each engine. Faster cores should speed up one request, which we have not measured.
• What comes next. This server will join our servers at CARouter Cloud.

Three engines on one card

An engine is the software that loads a model and answers requests. We tried three on one card: llama.cpp with its SYCL backend (its Vulkan backend was 2.8 times slower), Intel's build of vLLM (llm-scaler) and OpenVINO Model Server.

One request alone gets 67 to 70 tokens a second from llama.cpp, 59 to 62 from Intel's vLLM and 53 from OpenVINO. The GB10 gives 74.9, or 114.9 with the model's helper, which drafts the next tokens for the model to check.

Here the helper did not pay. On Intel's vLLM it gained 6% for one request and capped the total at 135 to 180 tokens a second.

With many requests at once the order changes. llama.cpp does not batch: two requests in one server are slower in total than one. Intel's vLLM writes 300 tokens a second in total with 8 requests and about 410 from 16, and OpenVINO 212 and 256.

llama.cpp accepts a request of up to 262 144 tokens, the model's whole window, one request at a time. Intel's vLLM shares its window between the requests in progress, and with the whole model loaded it has room for about 95 000 tokens. Memory is the tight side: of each card's 23.9 GiB, the 4-bit weights take 19.2.

We continued with Intel's vLLM, which does the most work with many requests at once.

Two cards, one engine each

We ran one vLLM engine on each card and measured both at the same instant. Each card gives what it gives alone: 409 tokens a second with 16 requests, so 818 in all and about 26 for each request.

With 8 requests per card the total is 558, about 35 each. The GB10 writes 342 in total with 16 requests, or 479 with its helper, which is 21 and 30 each.

So this machine is not faster than the GB10 for one request. It adds capacity: with many requests at once it does more work in total, and each request still gets about 35 tokens a second.

What this generation of processor allows

The server is built on a platform from 2018: an Intel Core i9-9900K, a 9th-generation processor with 8 cores and 16 threads, on an ASUS PRIME Z390-A motherboard with 32 GB of memory. It runs Ubuntu 24.04 on Linux kernel 7.0.

Intel's vLLM can spread one model over two cards, which must then exchange memory directly (peer-to-peer DMA). Between two cards on separate root ports, as here, the Linux kernel only allows that when the platform's host bridge is on its list of known-good ones.

The host bridge of this processor generation is not on that list, in kernel 7.0 as in the latest mainline we read. AMD Zen platforms and Xeon platforms are. So on a stock kernel, two cards here are two engines of 24 GB each, and not one pool of 48 GB.

The B60 is a PCIe 5.0 card with 8 lanes, and on this motherboard each card links at PCIe 3.0. That did not limit generation speed in our tests. It would matter for loading the model and for any traffic between pooled cards.

One processor thread paces each engine. From 1 to 16 simultaneous requests, each vLLM engine kept exactly one thread fully busy, while the 16 threads were 13% busy in all. Eight cores were enough for two engines.

From 2 to 8 simultaneous requests, each request keeps the same speed, about 35 to 38 tokens a second. That is the signature of a fixed cost per step on the host, and not of work on the card, which only saturates at about 16 requests.

Pooling the two cards anyway, as a test

To learn whether the motherboard itself was the obstacle, we lifted that one check. A small test-only kernel module made the kernel accept this server's host bridge. It is not a rebuilt kernel, it needs Secure Boot off, and a reboot removes it.

The pool then worked. A small model, Qwen3-0.6B, spread over both cards gave 6 of 6 answers identical to one card and ran a benchmark, with no fault in the kernel's log. The hardware carries the traffic.

It was slower than one card, with the same model and settings:

• One request alone: 36.8 tokens a second pooled, against 44.2 on one card.
• Eight requests at once: 298.8 in total, against 356.6.
• Reading a 12 000-token prompt: 16 555 tokens a second, against 25 793.

The real limit is the server's own memory. Everything the two cards share, weights and cache alike, also takes its size there, and swap does not absorb it. A 29 GB model ran this machine's 32 GB out while it loaded.

We have not pooled Qwen3.6 35B-A3B itself.

What a newer platform would probably add

None of what follows is measured. It is what the figures above lead us to expect.

• A newer processor, with faster cores, should raise the speed of one request and of small batches.
• A platform on the kernel's list would pool the two cards without a test module.
• More memory would let the pool hold a large model: about as much again as what the two cards share.
• PCIe 5.0 slots would link the cards at the speed they support, which should help loading and the traffic between pooled cards.

Long prompts and images

Intel's vLLM reads a 12 000-token prompt at about 5 900 tokens a second, against about 6 400 for the GB10 and 590 to 700 for llama.cpp.

Reading slows as the prompt grows. On one card, one request alone with a prompt of 11 400 tokens is answered in 2.9 seconds, with 55 216 tokens in 18.6 seconds, with 87 923 tokens in 39.2 seconds and with 132 606 tokens in 78.8 seconds.

Images work on Intel's vLLM: a 1024 by 1024 image is about 1 050 tokens and takes under a second.

Loaded without the part of the model that reads images, the weights take 18.4 GiB instead of 19.2. That leaves just enough room on one card for the model's whole window of 262 144 tokens.

Heat: test with many requests

The second card arrived with a fan that did not turn, and it reached 93 to 96 °C under load. One request alone kept its speed, at 62.4 tokens a second against 61.5 once repaired. With 8 requests, the total fell from 298.8 to 98.2.

With the fan repaired, the cards peak at about 68 to 74 °C with both under full load. A thermal problem does not show with one request, so test with many.

What comes next

This server will join our servers at CARouter Cloud.

The vLLM Semantic Router project has released two families of open decision models: Decision 2.0, and this week Vela 2.0. Both answer typed questions about a text the way Jev and Kev do.

We ran three of them beside Kev on our two benchmarks, on an NVIDIA GB10, the machine that serves Kev-4B at /v1/decisions. They are Decision 2.0 Nox-4B, Vela 2.0 4B, and Vela 2.0 0.3B, which is small enough to run without a GPU.

This is the fifth article of the series. Its table holds every decision model we have tested so far, and its last sections say what each one is good for.

Read the first benchmark. Read the second, with Kev's launch. Read the third, on OpenJev. Read the fourth, on Clef and Strands Decider.

The short version

• On our requests. Vela 2.0 4B named the kind of task right on 96.6% of our 328 requests, against 93.0% for Decision 2.0 Nox-4B and 92.4% for Kev-4B. It is the most accurate trained model we have tested, one request behind OpenJev's untrained Qwen3.5-4B.
• Where Vela gains. It recognised every casual chat and 29 of the 32 agent steps. CARouter reads an agent step from the request itself, and scored that way Vela 2.0 4B is at 97.6% and Kev-4B at 97.3%, one request apart.
• Without a GPU. Vela 2.0 0.3B reached 85.7% on a processor alone, ahead of Kev-0.8B on a GPU at 80.2%. A short decision took 92 milliseconds.
• On public prompts. Vela 2.0 4B reaches 80% in English and 76% in French, and Nox-4B 79% and 78%. Kev-4B is at 76% and 70%.
• Confidence. On public prompts, Nox-4B was wrong while at least 90% sure on 0.7% of the English requests, against 3% for Kev-4B and 7% for Vela 2.0 4B.
• Speed. A three-question decision on a short request takes 99 milliseconds on Kev-4B, 194 on Nox-4B and 319 on Vela 2.0 4B.
• Memory. Vela 2.0 4B holds 23 GB of the GB10's 121, Kev-4B 14 and Nox-4B 11. The 0.3B holds 1.5 GB on the GPU.

What the newcomers are

Decision 2.0 is a family of six models, from 0.6 to 27 billion parameters. We ran its 4B model, Nox, released under the Apache 2.0 licence. It is a fine-tune of Qwen3.5-4B-Base, the model Kev-4B is built on, and it answers the three kinds of question Jev does: a choice, a yes or no, and a score. Nox-4B's weights.

Vela 2.0's larger models start from Decision 2.0. They are trained for what a router must know about a request before sending it anywhere: its subject, whether it is an attack, whether it carries personal information, whether an answer is backed by its sources. The announcement. Vela 2.0 4B's weights.

It adds two kinds of answer. A set question returns every label that applies, and a span question returns the exact words it found, with their position in the text.

Vela 2.0 0.3B is built differently. It is an encoder of 0.3 billion parameters, the kind of model that reads a text without being able to write one, and it runs on a processor. Its weights are under Apache 2.0, and its tokenizer keeps Google's Gemma terms of use.

Their authors say the specialisation has a price. On the Jev Decision Index, a public suite of general decision tasks, the model cards report about 32 for Vela 2.0 4B and about 44 for the Nox-4B it started from. We expected Vela to read our requests worse than Nox, and the opposite happened.

The two 4B models read up to 16,384 tokens and the 0.3B reads 8,192, questions included. Decision 2.0 refuses a request that is too long, and Vela 2.0 cuts it. None of our test requests reached those limits.

How we tested

The test machine is a GB10, the same model as the one that serves Kev, with 121 GB of memory shared between the processor and the GPU. Each model ran alone on it, one request at a time, through its authors' own code. Kev ran the way our server runs it.

The two benchmarks are those of the earlier articles. On the 1,020 public prompts, each model answers one question with nine kinds of task. On our 328 requests it answers three: the kind of task, how hard it is, and whether the answer will be long.

One script scored every model, the one of the previous article. Before scoring anything new, it recomputes the published figures of the earlier articles and stops if one differs.

The models of the earlier articles keep the figures we published. Kev-4B ran again on this test machine as a reference. It chose the same answer as last time for 1,016 of the 1,020 public prompts and 325 of our 328 requests, and its timings are within 2% of the last article's.

Both Vela models gave their model card's answers on the card's own example, to within 0.001. Decision 2.0 publishes no answer to compare with. It checks every one of its files against a manifest before loading, and it answered its card's example sensibly.

The 0.3B ran three ways: on the GPU, and on the processor alone through two inference engines. All three chose the same answer on every request, so its accuracy has one row and its speed has two.

Jev is a paid hosted service that we did not call again. Its figures on the public prompts are those of the first benchmark, and it has never been run on our requests.

What we found: accuracy

Vela 2.0 4B got 317 of our 328 requests right. OpenJev's Qwen3.5-4B got 318, Clef 310, Nox-4B 305 and Kev-4B 303.

We wrote the questions about our requests with Kev-4B's answers in front of us, on three fifths of the requests. On the 137 requests kept aside, Vela 2.0 4B missed one: 99.3%, against 98.5% for OpenJev, 94.9% for Nox-4B and 93.4% for Kev-4B.

On the public prompts the newcomers sit with the best of the earlier models. Vela 2.0 4B reaches 80% in English and Nox-4B 79%, around the hosted Jev at 81% and ahead of Kev-4B at 76%.

French costs Nox-4B one point there, 79% against 78%. That is the smallest drop of any trained model in the table. Vela 2.0 4B loses five points, like Jev, and Kev-4B six.

Vela 2.0 0.3B got 281 of our requests right, 85.7%. That is 18 more than Kev-0.8B and 5 fewer than Strands Decider, two models that need a GPU. On the public prompts it is level with Kev-0.8B, at 71% in English and 64% in French.

Where the differences come from

Vela 2.0 4B is perfect on seven of the nine kinds. It recognised all 30 casual chats, where Nox-4B recognised 28 and Kev-4B 26.

Agent steps are where it gains most. It recognised 29 of 32, against 20 for Nox-4B and 16 for Kev-4B. The latest message of an agent step is a tool's result, which reads like code or a web page.

It is also the one kind CARouter does not ask a model about. A request that carries tool definitions is an agent step, and our platform sees that the way it sees the language. Scored that way, Vela 2.0 4B names 97.6% of the requests, Kev-4B 97.3% and Nox-4B 96.6%.

Research is the one kind where Kev-4B leads the newcomers: 23 of 28, against 21 for Nox-4B and 20 for Vela 2.0 4B. The 0.3B recognised 6 and filed 14 of the others under chat.

What we found: confidence

A model that decides on someone's behalf should act only when it is sure. So we count how often each one is wrong while at least 90% sure.

On the public prompts, Nox-4B is confidently wrong on 0.7% of the English requests and on none of the French ones. Vela 2.0 4B is at 7%, between Kev-4B at 3% and Clef at 8%.

On our requests, Vela 2.0 4B gave 269 answers at that level and two were wrong. The 0.3B gave 172, Kev-4B 187 and Nox-4B 59, with none wrong.

Each model computes its confidence its own way, and Decision 2.0's is the strictest: it falls as soon as a second option gets some probability. Recomputed with Kev's formula, Nox-4B is that sure of 144 answers, and its confident mistakes on the English public prompts rise from 0.7% to 5%.

Difficulty and long answers

Both 4B newcomers rate difficulty better than Kev-4B. Nox-4B places a request within one level of our rating 94% of the time and Vela 2.0 4B 92%, against 88% for Kev-4B and 81% for the 0.3B. Clef still leads at 98%.

Ten of our 328 requests need a long answer. Nox-4B flagged all ten and five others. Vela 2.0 4B found eight and flagged eight others, and Kev-4B found seven and flagged no other.

The 0.3B cannot answer that question. It found four of the ten and flagged 154 requests that did not need a long answer.

Speed and memory on a GB10

We timed the three-question decision of the earlier articles on a new request each time, from a short request to a 4,000-token one.

Kev-4B stays the fastest of the 4B models, at 99 milliseconds on a short request and 631 on a long one. Vela 2.0 4B takes 319 milliseconds and 1.3 seconds. Nox-4B takes 194 milliseconds and 3.1 seconds.

Nox-4B is slow on long requests because it reads the request again for every question. It has an option that reads it once, which brings the long request down to 1.1 seconds and changed none of its answers on our two benchmarks.

Asked a single question, Nox-4B is the fastest 4B model here: 72 milliseconds on a short request, against 83 for Kev-4B.

The 0.3B is in another class. On the GPU it takes 11 milliseconds on a short request and 243 on a long one, in 1.5 GB.

On the processor alone, using ten of its twenty cores, it takes 92 milliseconds on a short request, 581 on a 1,000-token one and 3.9 seconds on a 4,000-token one. In our third article, the fastest model on a GB10's processor was Kev-0.8B, at 1.3 seconds on a short request, with the cores shared with other work.

Vela 2.0 4B holds 23 GB, because its authors' code keeps its weights as 32-bit numbers. Kev-4B holds 14 GB and Nox-4B 11. On a processor, the 0.3B holds 4 GB of ordinary memory.

These timings are the best case for every model: alone on the machine, with no network in between. In production, Kev-4B shares its GB10 with other models and answers through an API, and the same decision takes 143 milliseconds and 1 second.

What each model is good for

A decision model answers a closed question about a text and says how sure it is. After five articles, here is where each model of the table fits.

• Choosing a model for each request. This is what our 328 requests measure. OpenJev's Qwen3.5-4B, Vela 2.0 4B and Clef read them best. Kev-4B is one request behind Vela once agent steps are read from the request, and it answers three times sooner.
• Decisions inside your own application. This covers sorting support tickets, sending a message to the right team, checking a condition before an action and rating urgency. Kev-4B does it today at /v1/decisions. Nox-4B fits where French matters or where a confident mistake is the costly one.
• Decisions without a GPU. Vela 2.0 0.3B answers on a processor while a caller waits, as long as the request is short. It is reliable on clear-cut kinds such as code, translation and extraction, and not on research or on long answers. Laya, in our first article, also answered in a fraction of a second on a processor, and it named 58% of the English public prompts right, against 71% here.
• Decisions beside a large model. When the GPU's memory belongs to a language model, the small ones matter: the 0.3B in 1.5 GB, Kev-0.8B and Strands Decider in 4 GB.
• Checking a request before it leaves. Vela 2.0 was trained to find personal information, prompt attacks, unsafe content and unsupported claims in the same read as its other questions. We tested none of that here. On CARouter, the PII filter and the content filter do this work with two separate models.
• Images and video. Clef is the only model here that reads them, and we tested text only.
• Nothing to run. Jev is a hosted service: there is no model to serve, and every request leaves your infrastructure.

What this does not prove

• This is one run per model on two sets of requests. A kind holds 20 to 66 of our requests, and on the smallest, one mistake moves the figure by five points.
• We wrote and labelled the 328 requests, and we tuned their questions on Kev-4B. Both benchmarks ask one thing: what kind of work a request is. On general decision tasks, Vela 2.0's own authors measure it below Decision 2.0.
• We did not test what Vela 2.0 was built for. Nothing here says how well it finds personal information, prompt attacks or unsupported claims, in English or in French.
• The new models ran through the code their authors published. Decision 2.0's fastest path is written for AMD GPUs and did not apply on the GB10's NVIDIA GPU, and Kev's serving code is tuned for this machine. The time gaps say as much about the software as about the models.
• The 0.3B was timed on the GB10's processor, an Arm chip with twenty cores. Another processor will give other timings.
• Jev was not called again. Its figures come from the first benchmark, and it was never run on our requests.

What this means for CARouter

Kev-4B stays behind /v1/decisions for now. Scored the way our platform reads a request, Vela 2.0 4B is one request ahead of it, for three times the wait and 8 GB more memory on a machine that serves other models.

Vela 2.0 4B is still the result of this article. It is the first trained model to read our requests about as well as the Qwen3.5-4B nobody trained, and it could answer routing questions and safety questions in one read. Before that changes anything, we have to test the safety half ourselves, in both languages.

Nox-4B is two requests ahead of Kev-4B, steadier in French and better at difficulty and long answers. It is also twice as slow on our three questions, which is not enough to switch.

The 0.3B changes one assumption. In our third article we concluded that a processor was no place for Kev or OpenJev while a caller waits. The 0.3B answers there in a tenth of a second on a short request, and it is a candidate wherever a deployment has no GPU to spare.

Every measurement behind this article is kept, and we will share it with anyone who asks.

Mistral released Mistral Large 4 on 6 October. We put it beside GLM-5.3, the flagship of China's Z.ai, on 148 hard coding problems, to see whether Europe's best model still keeps up with the best open models from elsewhere.

Published the same day: Germany's Kolibri-1 on a GB10.

The short version

• The score. Mistral Large 4 solved 56 of the 148 problems. GLM-5.3 solved 62 and its smaller sibling GLM-5.3-Flash solved 64. A gap that small is within what chance produces.
• The effort. Mistral Large 4 wrote 0.18 million tokens to answer. GLM-5.3 wrote 3.14 million, seventeen times as many.
• The bill. At list price, the 148 answers cost $0.41 from Mistral Large 4, $14.58 from GLM-5.3 and $1.03 from GLM-5.3-Flash.
• The verdict. Europe is still in the game. Its flagship is level with China's on this test, and an answer from it costs far less.

The three models

• Mistral Large 4. It is French, holds 1.05 trillion parameters and reads 49 billion of them per token. It answers directly unless asked to reason first, and we ran it both ways.
• GLM-5.3. It is Z.ai's flagship, at roughly 750 billion parameters. It reasons before every answer by default.
• GLM-5.3-Flash. It is the smaller sibling, at 320 billion parameters and 18 billion read per token, priced about nine times lower.

The test

We used BigCodeBench-Hard, a set of 148 problems. The model writes a Python function, and the function is run against tests. Each model answered each problem once, with its maker's default settings.

Mistral served Mistral Large 4 itself. Independent hosts served the two GLM models.

What came out

The models mostly agree. Mistral Large 4 and GLM-5.3 gave the same result on 126 of the 148 problems. Mistral solved 8 that GLM-5.3 missed, and GLM-5.3 solved 14 that Mistral missed. Chance alone produces a split like that about three times in ten.

The effort is where they differ. GLM-5.3 reasons at length before it answers, and on some problems it worked through the examples by hand for tens of thousands of tokens. A typical problem took Mistral Large 4 15 seconds and GLM-5.3 about a minute and a half.

Reasoning did not help Mistral Large 4. It solved 56 problems either way, and reasoning cost four times the tokens.

GLM-5.3-Flash did as well as GLM-5.3 for a fourteenth of the cost. On this test, the flagship's extra size bought nothing we could measure.

The test is hard for all of them. 66 of the 148 problems defeated every run, and 42 were solved by every run.

What we cannot tell you

• Whether six problems is a real gap. Settling that would take several times as many problems.
• How the models compare outside coding in English. We did not test French, German, images or long agentic tasks.
• How the GLM models behave on Z.ai's own service. We did not verify how each independent host runs the weights.
• Where an American model would land. We did not run one.

Is Europe still in the game?

On this evidence, yes. France's flagship stands level with China's on hard coding problems, and it gets there in a fraction of the tokens.

That matters to us for a practical reason. CARouter routes to Canadian providers, and open weights are what let a Canadian provider host a model. Mistral describes Mistral Large 4 as open-weight, and its weights were not yet published when we ran this test.

Aleph Alpha released Kolibri-1 on 3 October: an open-weight model from Germany, under the Apache 2.0 licence, built for German and English. It holds 78 billion parameters and reads about 3.5 billion of them for each token it writes.

That last figure is why we looked at it. Qwen3.6 35B-A3B, the model behind CARouter Cloud's coding endpoint, reads about 3 billion parameters per token out of 35 billion. The two do a similar amount of work per token, and one of them is more than twice the size. We ran both on the same GB10 to see what the extra size costs and what it buys.

An earlier article: why we serve Qwen3.6 35B-A3B.

The short version

• Memory. At 4 bits, Kolibri-1's weights take 42.7 GiB against 21.9 GiB for Qwen. As Aleph Alpha ships it, in FP8, they take 73.5 GiB and the model holds 103 GiB of a 121.6 GiB machine.
• Speed. One request alone gets 53 tokens a second from Kolibri-1 and 117 from Qwen. With sixteen requests at once the totals are 290 and 486.
• Coding. On our 50 coding problems, Kolibri-1 solved 29 to 32 depending on the build, and Qwen solved 33. Fifty problems cannot separate those scores.
• Long conversations. Kolibri-1 remembers them at half the cost: the same 28 GiB holds 9.9 conversations of 262,144 tokens against 4.8 for Qwen.
• What we decided. We are not adding Kolibri-1 to CARouter Cloud's machine. It would take twice the memory to answer coding requests about as well, at half the speed.

Two models that read the same amount per token

Both models are mixtures of experts. Each layer holds hundreds of small specialist networks, and each token is sent to a handful of them. All of them must sit in memory, and only the chosen few are read.

Kolibri-1 keeps 384 experts per layer and uses 6 of them per token. Qwen3.6 35B-A3B keeps 256 and uses 8. The work per token is close, so the speed should be close, and the memory should not be.

Which Kolibri-1 we ran

Aleph Alpha publishes Kolibri-1 in FP8, which is 8 bits per weight. We serve our own models at 4 bits, and Aleph Alpha publishes no 4-bit build, so we took two that the community made in the first days:

• The most downloaded one, a Q4_K_M file in the GGUF format. It runs on llama.cpp, with a patch its author ships because llama.cpp does not know this architecture yet.
• An NVFP4 build. It runs on vLLM, the software we serve with, in the same 4-bit format as our Qwen. This is the like-for-like comparison.

We also ran the FP8 release itself, to tell the model apart from its compression. Each model ran alone on the machine, with its maker's default settings, reasoning on.

Memory

Kolibri-1 at 4 bits needs 21 GiB more than Qwen, all of it weights. On a machine that also serves embeddings, a reranker and our decision model, that is the room those other models live in.

The FP8 release fits, and nothing else does. It left 10 GiB free on the whole machine.

Kolibri-1 wins the other half of the memory question. A model also has to remember each conversation in progress, and most of Kolibri-1's layers only look at the last 512 tokens. From the same 28 GiB it holds 2.59 million tokens of conversation against 1.25 million for Qwen.

Speed

Qwen is between 1.6 and 2.2 times as fast at every load. Part of that is a feature Kolibri-1 does not have: Qwen ships with a small helper that drafts the next tokens for the main model to check, which makes it half as fast again on this machine. Without the helper Qwen wrote 75 tokens a second alone, still ahead of Kolibri-1's 53.

A typical answer to one of our coding problems took Qwen 37 seconds for one user alone. The same problem took Kolibri-1 about 91 seconds.

Reading the prompt is a tie. Both models take in a 12,000-token prompt at about 6,400 to 6,800 tokens a second on vLLM.

The GGUF build is a different story on a different engine. Alone it is the fastest Kolibri-1, at 64 tokens a second. It shares the machine badly, with irregular totals between two and eight requests, and it reads a prompt three times slower. Going from 8 bits to 4 bits also made Kolibri-1 faster on vLLM, by up to 80% under load.

Quality

We use the same 50 BigCodeBench problems for every model we consider: the model writes a Python function, and the function is run against tests. Qwen solved 33, as it did on our own machine in August.

Kolibri-1 solved 32 in the Q4_K_M build, 29 in NVFP4 and 31 in FP8. The models agree on most problems: Q4_K_M and Qwen gave the same result on 47 of the 50. None of these gaps is statistically meaningful on 50 problems.

Kolibri-1 thinks longer to get there. It wrote between 1.3 and 1.7 times as many tokens as Qwen for the same 50 answers, and tokens are what a client pays for and waits for.

We also hid one sentence in documents of up to 220,000 tokens and asked for it back. Every model found it every time it finished its answer. Four times out of forty-eight, a model spent its whole answer allowance on reasoning first: Qwen once, Kolibri-1 three times.

Aleph Alpha's own published results point the same way. They put Kolibri-1 slightly ahead of Qwen3.6 35B-A3B on one coding benchmark and behind it on another.

What we cannot tell you

• How Kolibri-1 does in German. It is the model's first language and its main argument, and we did not test it. We did not test French either.
• Whether a gap of a few problems is real. Detecting a difference that small would take several hundred problems.
• How an official 4-bit build would behave. The two we ran were made by the community, without Aleph Alpha's involvement.
• What Kolibri-1 does beyond 262,144 tokens. Aleph Alpha validates it up to a million.
• How it scores with more reasoning. We left the reasoning effort at its default, and Aleph Alpha's published results use the highest setting.

What this means for CARouter

CARouter Cloud's machine keeps Qwen3.6 35B-A3B. For coding in English, Kolibri-1 gave us about the same answers for twice the memory and more than twice the wait.

That is a verdict on one machine and one use. Kolibri-1 is an Apache 2.0 model trained from scratch in Europe, it handles long conversations cheaply, and its German is the part we did not measure. A workload in German, or a provider with more memory per machine, would weigh these numbers differently.

Five weeks ago we put a faster model behind CARouter Cloud's coding endpoint. It came with a built-in speed feature that we had not switched on. Now it is on: the same model, on the same machine, at the same price, writes about one and a half times faster. The change was one line of configuration.

The previous article: why we now serve Qwen3.6 35B-A3B instead of Qwen3.8 27B.

The short version

• What a client sees. 115 tokens a second instead of 75. A long coding answer arrives in 41 seconds instead of 63.
• What we changed. One line. It switches on a small helper already inside the model file, which drafts the next three tokens for the main model to check.
• When the machine is busy. With sixteen requests at once, 479 tokens a second in total instead of 342.
• What it costs. 2.3 GiB of memory out of 121. No new hardware and no new download.
• How we tested it. On a rented identical machine, for 1.03 US dollars. Production restarted once, for six minutes.

A helper that drafts, a model that checks

A language model writes one token at a time, a token being a word or a piece of one. Each token costs one full pass of the model.

With speculative decoding, a small helper drafts the next few tokens and the large model checks them all in one pass. It keeps the ones it agrees with and rewrites the first one it does not. The large model has the last word on every token, so the helper saves time without lowering quality.

Qwen3.6 35B-A3B ships with such a helper inside its own file, called a multi-token-prediction head. We had missed it when we evaluated the model in August.

What we measured

Same machine, same model file, same memory budget. We measured, restarted the endpoint once with the helper on, and measured again the same evening.

The helper drafts three tokens at a time and about seven in ten are accepted. Each pass yields about three tokens instead of one and costs about twice as much, which makes one and a half times faster.

The gain is 1.53 times for one user and 1.40 times for sixteen. Across everything we measured it stays between 1.35 and 1.58 times.

What that means for someone waiting

A short reply arrives two seconds sooner and a long coding answer twenty-two seconds sooner. With eight people on the endpoint, each gets that long answer in 99 seconds instead of 145.

A coding agent calls the model hundreds of times for one task. A third less waiting on every call adds up.

What it costs

The endpoint holds 2.3 GiB more memory, which the machine had to spare. Reading the prompt is 1% slower.

It also remembers a little less conversation at once: 4.8 conversations of the maximum length instead of 5.5. Our traffic is far below that, so we changed no limit.

How we tested it without switching anything off

Every restart of the coding endpoint is six minutes during which it answers nobody. Trying each setting in production would have meant four of them.

So we rented a machine with the same chip, by the hour, and ran our own software on it unchanged. We tried the helper with one, two and three tokens. Three won, and the evening cost 1.03 US dollars.

Then we changed one line in production and restarted once. The rented machine had predicted the result to within a few percent.

What we cannot tell you

We did not run the coding benchmark again. The large model still decides every token, and we checked that conversation, reasoning, tool calls and images all still work.

We cannot show two identical answers. With all randomness switched off, this software does not always repeat itself, with the helper or without it. We found no difference that the helper causes, and we cannot prove there is none.

One faster option remains, a computation routine written for this chip. It has open defects in the software we depend on, so we are not running it.

What is running now

Qwen3.6 35B-A3B with its prediction helper, on the same GB10, under the same name, qwen3.6-35b-a3b, at the same price. Context, images, tools and reasoning are unchanged.

This policy applies only to models we host ourselves through CARouter Cloud. Models served by other providers keep their published rates, converted to Canadian dollars, plus your plan's markup. Their pricing is unchanged by this policy.

We want prices that are logical for the models available today and for those we add later. We use a market reference to position an offer, cost estimates to assess its sustainability, and explicit launch decisions while demand grows.

What we mean by market reference

The market reference benchmarks alternative hosting of the same model weights. Our default starts with the mean price of eligible OpenRouter offers, converted to Canadian dollars, then adds a 10% sovereignty markup for Canadian-owned, Canadian-served inference. The benchmark covers model hosting; our routing and platform services are priced separately.

We estimate that markup by giving equal weight to selected pricing observations from Google Cloud, OUTSCALE, Microsoft Azure and AWS: (5% + 20% + 0% + approximately 15%) / 4 = approximately 10%.

This is our selected-offer benchmark. The packages have different guarantees, and the average excludes taxes, separate support subscriptions and additional security services. Azure's zero refers to its base surcharge, not the total cost of a sovereign deployment. The AWS estimate comes from tecRacer's January 2026 calculator comparison with Frankfurt, after currency conversion; we have not independently refreshed it. We adopt 10% as an initial positioning assumption and review the selection quarterly.

These infrastructure and control-package prices do not establish equivalent certifications or an equivalent Canadian routing service. Selecting different packages would change the average: using a Google regulatory package at 20% would give approximately 13.75% instead. We use its regional data-and-support package as our starting assumption.

Comparisons account for context, performance, caching, availability and residency. When an alternative already provides comparable sovereignty, we use its rate without adding that premium again.

The sovereignty markup is part of the CARouter Cloud inference base rate. Your plan's markup applies separately for routing, billing and platform operations.

Where no comparable offer exists

We estimate a starting price from the capacity a model reserves and the work it performs: memory, measured throughput, hardware, hosting and electricity. We use a common contribution target, then check the result against alternative ways to perform the task.

A cost per token depends on how much capacity is used. Our initial reference assumption is 10% utilization. We also examine 5% and 20% scenarios at the same chosen prices. These are planning assumptions; actual costs per token depend on the workload and measured utilization.

Investing while demand grows

As a startup, we can choose a launch price below estimated sustainable serving costs. We record the reason, set a review date and budget the unrecovered capacity cost. A quiet machine does not automatically make your next request more expensive.

An expensive deployment can call for better batching, a smaller memory reservation or different hardware. We assess those options alongside pricing. The service must remain useful at the price customers pay.

What this changes for your bill

We price fresh input, cached input and output together. Reusing a cached prompt avoids repeated prompt processing, so confirmed cache hits receive a discount. Generating an answer still uses compute.

The effect depends on your workload. Repeated-context conversations and agents can benefit from the cache discount, while workloads that generate long answers with little caching can cost more. We check representative requests and existing usage before changing rates.

Example: Qwen3.6 35B A3B

In our October 1, 2026 snapshot, we convert the mean OpenRouter rates from USD to CAD at 1.4243, then apply the 10% sovereignty markup and round to four decimals. These illustrative rates are in CAD per million tokens, before the plan markup.

For 12,000 prompt tokens and 100 output tokens, this gives CAD $0.003049 without caching, or approximately CAD $0.001554 when 11,000 prompt tokens are confirmed cached, before the plan markup and billing rounding.

We also check sustainability: at 10% utilization in our colocation scenario, estimated serving costs per million tokens are CAD $0.0804 for fresh input, $0.0031 for cached input and $1.4897 for output. These estimates exclude company overhead; any uncovered capacity cost at lower demand is part of our launch budget.

See the current rates in the documentation.

Quarterly reviews, announced changes

Every three months we review market references, serving costs, performance and demand. A review can leave prices unchanged. It does not automatically reprice your requests.

We announce increases to affected customers before they take effect, with an effective date and an estimate based on their usage. Decreases can take effect immediately. The documentation is generated from the catalog used for billing, so it remains the place to check the applicable rates.

Two new open decision models have just been released. Cloudflare published Clef and its smaller sibling Clef-flash, and Strands Labs published Strands Decider. All three answer typed questions about a text the way Jev and Kev do.

We ran them beside Kev on our two benchmarks, on an NVIDIA GB10, the machine that serves Kev-4B at /v1/decisions. We measured how often each one is right, how often it is wrong while sure, how long a decision takes and how much memory it holds.

Read the first benchmark. Read the second, with Kev's launch. Read the third, on OpenJev.

The short version

• On our requests. Clef named the kind of task right on 94.5% of our 328 requests, against 92.4% for Kev-4B, 89.6% for Clef-flash and 87.2% for Strands Decider.
• Where Clef gains. The gap is seven requests, and it comes from agent steps: Clef recognised 28 of 32 and Kev-4B 16. CARouter reads an agent step from the request itself, and scored that way Kev-4B is at 97.3% and Clef at 95.7%.
• On public prompts. The three newcomers reach 80 to 82% in English, level with the hosted Jev and ahead of Kev-4B at 76%. In French, Clef-flash leads at 78% and Strands Decider falls to 68%.
• Confidence. On public prompts, Clef and Clef-flash were wrong while at least 90% sure on 8% and 9% of requests, against 12% for Jev, 3% for Kev-4B and 0.2% for Strands Decider.
• Speed. A three-question decision on a short request takes 101 milliseconds on Kev-4B, 108 on Strands Decider, 173 on Clef-flash and 562 on Clef.
• Memory. Clef holds 52 GB of the GB10's 121, Clef-flash 18, Kev-4B 14 and Strands Decider 4.
• The model nobody trained. OpenJev's Qwen3.5-4B, run again on the same machine, still leads on our requests at 97.0%, in 192 milliseconds and 9 GB.

What the newcomers are

Clef is Cloudflare's family of decision models, released under the Apache 2.0 licence with its weights on Hugging Face. Clef is built on Qwen3.8-27B and Clef-flash on Qwen3.5-9B. Cloudflare's announcement. Clef's weights.

Cloudflare added a small head on top of each chat model. It reads the model's last layer and scores every option of every question in one pass. Clef takes the same request as Jev, and it can also read images and video, which we did not test.

Strands Decider comes from Strands Labs, the experimental side of the Strands Agents project, also under Apache 2.0. It has 1.9 billion parameters: an adapter and a small head on Qwen3.5-2B-Base. That is how Kev is built, on a base that sits between Kev-0.8B and Kev-4B. Source code.

The three read different lengths of text. Strands Decider's window is 4,096 tokens, Kev's is 8,192 as we serve it, and Clef's is 16,384 as released. Clef and Strands Decider cut a text that is too long and keep its beginning, where our Kev engine refuses the request. None of our test requests reached those limits.

How we tested

The test machine is a GB10, the same model as the one that serves Kev, with 121 GB of memory shared between the processor and the GPU. Each model ran alone on it, one request at a time, through its authors' own code. Kev ran the way our server runs it.

The two benchmarks are those of the earlier articles. On the 1,020 public prompts, each model answers one question with nine kinds of task. On our 328 requests it answers three: the kind of task, how hard it is, and whether the answer will be long.

One script scored every model, the one of the previous article. Before scoring anything new, it recomputes the published figures of the earlier articles and stops if one differs.

We checked the setup three ways. Strands Decider gave its model card's answers on the card's own example, to within 0.004. OpenJev gave its authors' published answer on all 144 of its test decisions. Kev-4B on the test machine chose the same answer as in production for 1,015 of the 1,020 public prompts and 327 of our 328 requests.

Cloudflare publishes no answer-by-answer results for Clef, so we could not check it the same way. It loaded as its card describes and gave sensible answers to the card's examples.

Jev is a paid hosted service that we did not call again. Its figures on the public prompts are those of the first benchmark, and it has never been run on our requests.

One thing changed since the OpenJev article. We reworded two options of the question about the kind of task, chat and research, and an agent step now shows the user's task beside the tool's result. Every model here answered the new wording.

That article's figures on our requests answered the old wording and do not compare with these. With the new one, Kev-4B went from 88.7% to 92.4%, and OpenJev's Qwen3.5-4B from 95.7% to 97.0%.

What we found: accuracy

On our requests, Clef is the most accurate of the trained models: 310 right out of 328, against 303 for Kev-4B. Clef-flash got 294 and Strands Decider 286.

On the public prompts the order changes. Clef-flash, Clef and Strands Decider reach 82%, 82% and 80% in English, around the hosted Jev at 81%, and Kev-4B stays at 76%.

French separates them. Clef-flash keeps 78% and Clef 76%, as Jev does. Strands Decider drops to 68%, below Kev-4B at 70%. On our requests it loses less: 89% in English and 85% in French.

We wrote the questions about our requests with Kev-4B's answers in front of us, on three fifths of the requests. On the 137 requests kept aside, the order is the same: OpenJev 98.5%, Clef 95.6%, Kev-4B 93.4%, Clef-flash 91.2% and Strands Decider 88.3%.

Where the differences come from

Every model is perfect on code, math, business writing and translation. The differences sit in three kinds of request: agent steps, casual chat and research.

Agent steps are where Clef gains. It recognised 28 of 32, where Strands Decider recognised 19, Kev-4B 16 and Clef-flash 14. The latest message of an agent step is a tool's result, which reads like code or a web page.

It is also the one kind CARouter does not ask a model about. A request that carries tool definitions is an agent step, and our platform sees that the way it sees the language. Scored that way, Kev-4B names 97.3% of the requests, Clef 95.7%, Clef-flash 95.1% and Strands Decider 91.2%.

On research, Strands Decider and Kev-4B do better than both Clef models: 89% and 82%, against 71% for Clef and 50% for Clef-flash, which files a research request under extraction or summarization. Strands Decider's weak kinds are chat, at 57%, and extraction, at 63%.

What we found: confidence

A model that decides on someone's behalf should act only when it is sure. So we count how often each one is wrong while at least 90% sure.

On the public prompts, Clef and Clef-flash are confidently wrong on 8% and 9% of the English requests. That is better than Jev at 12% and worse than Kev-4B at 3%. Strands Decider was confidently wrong once in 540 requests.

On our requests almost nobody is wrong while sure, and the models differ in how often they are sure. Clef-flash gave 227 answers at that level and two were wrong. Clef gave 213, Kev-4B 187 and Strands Decider 131, with none wrong.

Each model computes its confidence its own way. Kev and Strands Decider correct it for the number of options, and Clef reports the probability of its first choice. Recomputed with Kev's formula, Clef is that sure of 197 answers and Clef-flash of 218, and the order does not change.

Difficulty and long answers

Clef and Clef-flash are the best at rating difficulty. Both place a request within one level of our rating 98% of the time, against 90% for Strands Decider and 88% for Kev-4B. On the exact level they reach 46%, against 41% for Kev-4B.

Ten of our 328 requests need a long answer. Clef flagged all ten and two others, and Kev-4B flagged seven and no other. Clef-flash also found the ten and flagged 13 others. Strands Decider found eight and flagged 58 others.

Speed and memory on a GB10

We timed the three-question decision of the earlier articles on a new request each time, from a short request to a 4,000-token one.

Kev-4B is the fastest of the larger models, at 101 milliseconds on a short request and 639 on a long one. Strands Decider, half its size, takes 108 and 820 milliseconds. Kev's serving code prepares its computations ahead of time, and Strands Decider's reference code does not.

Clef-flash takes 173 milliseconds and 1.4 seconds. Clef takes 562 milliseconds and 4.4 seconds, five to seven times longer than Kev-4B.

Cloudflare reports a median of 209 milliseconds for Clef and 39 for Clef-flash in its own tests. On the GB10, a one-question decision on the public prompts took a median of 428 and 141 milliseconds. A GB10 is a desktop machine, not a datacenter GPU.

Memory matters more on this machine, because the processor and the GPU share one pool. Clef holds 52 GB, more than two fifths of it, and takes about six minutes to load. Clef-flash holds 18 GB, Kev-4B 14 GB and Strands Decider 4 GB, the same as Kev-0.8B.

These timings are the best case for every model: alone on the machine, with no network in between. In production, Kev-4B shares its GB10 with other models and answers through an API, and the same decision takes 143 milliseconds and 1 second.

The model nobody trained

The previous article left a question open: how fast OpenJev's Qwen3.5-4B would be on the GPU that serves Kev. OpenJev trains nothing and asks an ordinary chat model to pick an option letter. We ran it again on the same machine, with the reworded questions.

It is still the most accurate model on our requests: 97.0%, the same in English and in French, with every casual chat and 30 of the 32 agent steps recognised. With agent steps read from the request, it is at 97.6% and Kev-4B at 97.3%, one request apart.

It is also still the most confident. It gave 292 answers at 90% confidence or more, and four were wrong. On the public prompts it is confidently wrong on 19% of the requests, the highest figure in the table.

On the GB10's GPU, its three-question decision takes 192 milliseconds on a short request and 1.1 seconds on a long one, in 9 GB. That is about twice Kev-4B's time on a short request, and faster than Clef.

What this does not prove

• This is one run per model on two sets of requests. A kind holds 20 to 66 of our requests, and on the smallest, one mistake moves the figure by five points.
• We wrote and labelled the 328 requests, and we tuned their questions on Kev-4B. Both benchmarks ask one thing: what kind of work a request is. Cloudflare's own tables cover some forty other tasks, from tool selection to contract reading, and put Clef ahead of Jev and Kev-9B on most of them.
• Clef and Strands Decider ran through the code their authors published. Kev's serving code is tuned for speed, and so, presumably, is what Cloudflare runs in production. The time gaps say as much about the software as about the models.
• We tested text only. Clef also reads images and video, and no other model here does.
• Jev was not called again. Its figures come from the first benchmark, and it was never run on our requests.

What this means for CARouter

Kev-4B stays behind /v1/decisions. Clef is more accurate on our requests only where our platform already knows the answer. On a machine that serves other models, it would take 52 GB and five times the wait.

Clef-flash is the more practical of the two. It leads on the public prompts in both languages and rates difficulty better than Kev, for 18 GB and under 200 milliseconds on a short request. It also reads half of our research requests as something else.

Strands Decider is the one to watch at the small end. In the 4 GB that Kev-0.8B holds, it is seven points more accurate on our requests and eleven on the English public prompts, and it was never wrong while sure on ours. It is also almost five times slower than Kev-0.8B, and weaker in French.

The open question is still the one of the last article. A Qwen3.5-4B that nobody trained reads our requests best, and we now know what it costs on the GPU: 192 milliseconds and 9 GB. Its confidence on other kinds of requests still needs checking, and the full selection test still has to be run with it.

Every measurement behind this article is kept, and we will share it with anyone who asks.

Kev-4B is the decision model CARouter serves at /v1/decisions. We chose it on 1,020 public prompts, then measured it on 328 requests written like the work our customers send.

OpenJev takes another route and trains nothing. It asks an ordinary chat model to pick an option letter, then reads how likely each letter was. We ran it on both benchmarks, scored by the same script as Kev, to find out whether it would have been the better choice.

Read the first benchmark. Read the second, with Kev's launch.

The short version

• On our requests. OpenJev's Qwen3.5-4B named the kind of task right on 95.7% of our 328 requests, against 88.7% for Kev-4B, with the same score in English and in French.
• Kev's weak spots. It recognised casual chat 97% of the time and a single step of an agent's work 88% of the time. Kev-4B gets those right a third and half of the time.
• On public prompts. It led there too, at 79% in English and 81% in French, level with the hosted Jev. But it was confidently wrong on 19% of them, against 3% for Kev-4B.
• Confidence on our requests. It was at least 90% sure of 294 answers and wrong on two of them. Kev-4B was that sure of 183 answers and never wrong.
• The browser file. The 3 GB file openjev.com downloads scores as well as the full model. The two smaller models the site offers are far behind and often confidently wrong.
• Speed. On this laptop's GPU, the browser file makes a three-question decision in 267 milliseconds on a short request. On a server processor alone it takes 9 to 12 seconds, slower than both Kev models, where Kev-4B answers in 143 milliseconds on the GPU that serves it.

What OpenJev is

OpenJev is an independent open-source project, renamed SemIf by its author, with no tie to TypeSafe, the company behind Jev. Its code is under the MIT licence, and openjev.com runs it inside the browser. openjev.com · Source code.

It puts the text and the question into a prompt, lists the options as letters A, B, C, and asks the model to reply with one letter. The model never writes anything. OpenJev reads the probability the model gave each letter, and those become the probabilities of the answer.

Its best model is Qwen3.5-4B. Kev-4B was trained from the base version of that same model, so this test also asks a plain question: does Kev's training beat what Qwen's own instruction tuning already knows?

How we tested

We asked OpenJev exactly what we had asked Kev. On the public prompts, that is one question with nine kinds of task. On our requests, it is three questions: the kind of task, how hard it is, and whether the answer will be long.

One script scored every model. Before scoring anything new, it recomputes the figures of both previous articles from their raw answers and stops if a single one differs.

OpenJev ran with its own code, at a pinned version. As a check, we first ran its own 144 test decisions. Our setup gave the answer its authors published on all 144 of them, and the same 0.813 score.

We ran each OpenJev model twice: at full precision, where its published scores come from, and as the smaller file the website actually downloads. OpenJev takes one question at a time with a list of options, so a difficulty scale became five options and a yes-or-no question became Yes and No.

We also ran Kev-0.8B and both Laya models on our 328 requests, which only Kev-4B had seen. Jev and Kev-9B are missing there: Jev is a paid hosted service we did not call again, and Kev-9B does not fit on the laptop we used.

What we found: accuracy

On our requests, OpenJev's Qwen3.5-4B is the most accurate model we have measured: 95.7% at full precision and 97.3% as the browser file, against 88.7% for Kev-4B. English and French score the same.

On the public prompts, it matches Jev and passes Kev-4B by four points in English and eleven in French. French cost every model in the first benchmark five to eight points, and it cost OpenJev nothing.

The site's smaller models change the picture. MiniCPM5-2B, the default on a desktop, holds up on the public prompts but falls to 75% on ours. Qwen3-0.6B, offered for phones, trails every model except Laya.

Where the difference comes from

The gap sits in two kinds of request. OpenJev recognised casual chat 97% of the time and single agent steps 88% of the time. Kev-4B reads a general-knowledge question as research, and an agent's next step as code.

Kev-4B is better with documents: 93% against 82% on research and 100% against 85% on summaries. OpenJev tends to read a long research request as business writing. Both are perfect on code, math, business writing and translation.

What we found: confidence

A model that decides on someone's behalf should act only when it is sure. The first article measured how often a model is wrong while at least 90% sure, and here the two benchmarks disagree about OpenJev.

On our requests, OpenJev was that sure of 294 answers out of 328 and wrong twice, both times calling a research request business writing. Kev-4B was that sure of 183 answers and never wrong. A bar at 90% would let OpenJev decide nine requests in ten on its own, and Kev-4B just over half.

On the public prompts, OpenJev was confidently wrong on 19% of the English requests, against 3% for Kev-4B and 12% for Jev. Many of those labels are debatable.

The dataset files "How many U.S. Presidents have served in the military?" under brainstorming. OpenJev called it a factual question with 99.8% confidence, and we counted it wrong, as we did for every model.

Not all of them are the labels' fault. Counted by family of task, where such disputes stop mattering, OpenJev is still confidently wrong on 15% of the public prompts. Its confidence held on requests like ours, and it needs checking on anything else.

The browser file and the smaller models

The 3 GB file openjev.com downloads is a 4-bit copy of Qwen3.5-4B. It scored 97.3% on our requests and 80% on the public prompts in both languages, as well as the full model.

Two things change inside the browser. The page asks its question in its own words, and with them the same file scores 92.7% on our requests instead of 97.3%, still ahead of Kev-4B. It also gives the model a window of 2,048 tokens, and 14 of our 328 requests do not fit in it.

The two smaller models are overconfident as well as less accurate. MiniCPM5-2B was confidently wrong on 14% of our requests and Qwen3-0.6B on 24%, where Kev-4B and Kev-0.8B were never confidently wrong.

Difficulty and long answers

On the other two questions the two 4B models are close. OpenJev places a request on our exact difficulty level 36% of the time, against 41% for Kev-4B, and within one level 87% against 88%.

Neither is good at predicting a long answer. Only 10 of our 328 requests need one, and OpenJev flagged 4 of them against 3 for Kev-4B.

Speed and memory

OpenJev asks one question per pass over the text. Three questions take three passes, or one pass over the text and three short ones, which is the faster mode its code offers. Kev reads the text once and answers every question from that single pass.

On this laptop's GPU, the browser file of OpenJev's Qwen3.5-4B makes a three-question decision in 267 milliseconds on a short request and 2.2 seconds on a 4,000-token one. Kev-4B does not fit on this GPU, so the two 4B models were never timed side by side. On the first benchmark's datacenter GPU, Kev-4B took 58 milliseconds and 1 second.

Kev-4B does fit on a MacBook Air with Apple's M4 chip and 24 GB of memory, which its GPU shares with the processor. Through MLX, Apple's machine-learning library, which Kev supports, the same decision took 1.3 seconds on a short request and 14 seconds on a 4,000-token one.

Through PyTorch, the library our own server runs Kev with, it took 3.6 seconds on a short request and 20 seconds on a 1,000-token one. On a Mac, PyTorch has no fast version of Kev's recurrent layers and reads the text again for each question, and at 4,000 tokens it needed more memory than the Mac has. On six check prompts, both gave the same answers as the datacenter GPU.

The full Qwen3.5-4B needs 8.7 GB for its weights, more than this laptop's GPU holds. We measured its accuracy with part of it kept in system memory, and did not time it.

The browser file takes about 3 GB on the GPU, everything included. Kev-4B used up to 17 GB on the first benchmark's GPU, and holds under 9 GB on the machine that serves it.

On a processor alone

We timed the same three-question decision on two server processors, with no GPU in reach. One is a virtual machine with four AMD EPYC (Milan) cores and eight threads. The other is eight of the ten fast Arm cores of the GB10 machine that serves Kev, run at the lowest priority beside the models it serves. Kev ran the way our server runs it, reading the text once and then each question, and OpenJev ran its browser file in the faster of its two modes. The dotted lines are Kev on the GB10's GPU, on the same requests.

On a processor, OpenJev's 4B file is the slowest of the three. On the GB10's cores it took 8.9 seconds on a short request and 87 seconds on a 4,000-token one, against 6.1 and 53 seconds for Kev-4B and 1.3 and 15 seconds for Kev-0.8B. On the EPYC cores, OpenJev and Kev-4B were close, at 12 and 10 seconds on a short request, and Kev-0.8B took 1.8 seconds.

None of this works while a caller waits. On the GPU of the same machine, Kev-0.8B answers in 52 to 293 milliseconds and Kev-4B in 143 milliseconds to 1 second, about 25 to 50 times faster than the same model on its fastest processor.

Two details matter on a processor. The EPYC has no fast arithmetic for the 16-bit numbers Kev is stored in, so Kev-0.8B ran seven to eight times slower in that format than in 32-bit numbers. The GB10's Arm cores have that arithmetic, and there the two formats ran at about the same speed. The EPYC machine's eight threads are also only four cores: with four threads, Kev was about 20% slower.

On the GB10, moving a model to the processor would not free memory either, since its processor and GPU share one pool and a model takes the same room on both. Kev-4B also reads its weights as 32-bit numbers, about 16 GB, to fold its adapter in exactly. That does not fit beside the models the machine serves, so on its processor we timed Kev-4B in 16-bit with the adapter kept apart.

What this does not prove

• This is one run per model on two sets of requests. On a kind with 20 requests, one mistake moves the figure by five points.
• We wrote and labelled the 328 requests ourselves. On OpenJev's own 144 test decisions, Kev's model card reports the reverse, 0.896 for Kev-4B against 0.813 for OpenJev's Qwen3.5-4B, so the winner depends on the requests.
• The browser file ran through llama.cpp on the laptop's GPU, not inside a browser. Its accuracy carries over, but its speed does not: OpenJev's authors measured 3.3 seconds for one decision in Chrome on an RTX 3090.
• The laptop timings come from two laptops, and the processor timings from one EPYC machine and one GB10, each run once with five requests per length. The GB10's cores were shared with the models it serves.

What this means for CARouter

We chose Kev-4B because it was the most accurate open model on public prompts and the most honest about its doubts. On requests like the ones our customers send, an untrained Qwen3.5-4B reads them better, including the two kinds Kev-4B misses most.

Kev-4B stays behind /v1/decisions for now, on the GPU: a processor is no place for any of these models while a caller waits. Before changing anything, we will measure the prompted Qwen3.5-4B on the GPU that serves Kev, at the speed a customer would see, and run the full selection test that automatic model selection needs. Every measurement behind this article is kept, and we will share it with anyone who asks.

Kev 4B is now available on CARouter at /v1/decisions. It is a decision model: it reads a text and answers typed questions about it with a probability for each answer, the way TypeSafe's hosted Jev does, and it runs on our own hardware in Canada.

We measured it on 328 new requests in English and French, then used its answers to pick a model for each request among the chat models we serve. Here is what it gets right, what it costs, and what choosing a model this way is worth.

The short version

• On the 137 requests we kept aside from every design choice, Kev named the kind of task correctly 98.5% of the time, 98.6% in English and 98.4% in French.
• Kev gave 76 of those answers with at least 90% confidence, and not one of them was wrong.
• Picking a model per request with Kev's answers came within two requests of the best single model, at 25% less cost and 35% less waiting.
• A three-question decision takes 146 milliseconds on a short request and 1.2 seconds on a 4,000-word one.
• It costs CAD 0.0532 per million input tokens, and output is free.

What Kev does

A chat model answers in sentences that someone then has to read. Kev never writes anything. You send it a state, which can be a text or a JSON object, with up to 32 questions of three types: a choice among named options, a score on ordered levels, or a yes or no.

Each answer comes back with its probabilities and a confidence. Kev reads everything in one pass and generates nothing, so it cannot invent an answer that is not among your options.

Here is a support email sent to Kev with three questions about it.

curl https://carouter.ai/v1/decisions \
  -H "Authorization: Bearer $CAROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kev-4b",
    "state": "Hi, I was charged twice for invoice 2026-0417 this month. Can you refund the second charge before Friday? Our accounts close then.",
    "questions": {
      "intent": {"type": "choice", "instructions": "What is the customer asking for?",
                 "criteria": {"refund": "money back for a charge",
                              "invoice_copy": "a copy of an invoice",
                              "cancellation": "ending their subscription"}},
      "urgent": {"type": "noul", "instructions": "Does the customer need an answer within the week?"},
      "tone":   {"type": "score", "instructions": "How upset is the customer?",
                 "criteria": ["calm", "annoyed", "angry"]}
    }
  }'

Kev answered that the customer wants a refund, with 98% probability, that the matter is urgent, at 88%, and that the customer is annoyed rather than angry. It read 111 tokens to do so.

{
  "object": "decision",
  "model": "kev-4b",
  "answers": {
    "intent": {"type": "choice", "choice": "refund", "confidence": 0.9752,
               "probabilities": {"refund": 0.9835, "invoice_copy": 0.0077, "cancellation": 0.0088}},
    "urgent": {"type": "noul", "noul": 0.8772},
    "tone":   {"type": "score", "score": 1.1696, "confidence": 0.8253,
               "legend": {"0": "calm", "1": "annoyed", "2": "angry"},
               "probabilities": {"0": 0.0899, "1": 0.6507, "2": 0.2594}}
  },
  "usage": {"prompt_tokens": 111, "total_tokens": 111},
  "served_region": "CA-ON"
}

The request follows the shape of TypeSafe's SystemOne API, plus the model field every CARouter request carries. The state can run to 8,192 tokens. Past that, Kev refuses the request rather than cutting it, because it would keep the start of a long text and drop its end.

Same answers as in our research

Before opening Kev to customers, we replayed the 1,020 requests of our previous benchmark on the machine that now serves it. It chose the same answer as the test machine on 1,017 of them, and its accuracy matched to the third decimal in both languages.

The three differences were near-ties, under 45% confidence on either machine. The model you call is the model we measured.

A new benchmark, closer to real work

Public datasets do not contain what our customers send: a code review, a tax question about a Quebec invoice, an agent's next step after a tool result, a statute or a statistics release pasted whole. So we wrote 328 requests of that kind, half in English and half in French.

The French was written for a reader in Quebec or Ontario, not translated from the English. The long documents come from openly licensed sources such as federal statutes, Statistics Canada releases, Wikipedia and public-domain books. Every request records where its words come from.

We asked Kev the three questions an automatic model selector needs: what kind of task this is, how hard it is, and whether a good answer will run long. It saw the start of the instructions, the tool names, and the beginning and end of the latest message, or the user's task when that message is a tool result.

We wrote those questions and the routing rules on three fifths of the requests. Every figure below comes from the other two fifths: 137 requests that none of those choices saw.

What we found

Kev named the kind of task correctly 98.5% of the time on those 137 requests. It was right on every request in seven of the nine kinds, and missed one research request and one chat request.

Agent steps are read from the request itself. A request that carries tool definitions is one step of an agent's work, and the router sees that the way it sees the language and the length. Kev alone recognized half of them, because the latest message is a tool result that reads like code or a web page.

The line between a quick question and a research question is a judgment call, and the labels are ours. A different labeller could move those two figures by a few points.

Honest about its doubts

For a model that decides on someone's behalf, the number that matters most is how often it is wrong while sure of itself. Kev gave 76 answers with at least 90% confidence, and all 76 were right.

Every answer Kev gave with at least 60% confidence was right, and 86% of its answers reached that bar. Below it, a selector should send the request to a default model rather than trust the reading.

Difficulty and long answers

Kev places a request within one level of our difficulty rating 85% of the time. It is the least precise of the three readings, and our routing table does not use it yet.

A request that states its own length, such as at least 1,000 words or a complete module, is read as needing a long answer from its wording. Together with Kev, that caught 3 of the 5 held-out requests that need one, and long answers still need work.

Picking the model for each request

We then ran every request through the four chat models we serve from Canada: Qwen3.6 35B A3B and Qwen3 0.6B on our own hardware, and Augure's Tofino 3 and Rosedale 1. Each answer was graded by a test, an exact result or a rubric.

Code ran against its tests, while math and extraction were compared with the right answer. Prose was scored against a rubric by Qwen3.6 35B A3B with its thinking off, a judge that failed 2 of 122 correct answers and passed none of 122 wrong ones.

From the three fifths used for design, we wrote a routing table by kind of work. Tofino 3 takes agent steps, math, translation and chat, where it ties or leads in seconds and at a fraction of the price. Qwen3.6 takes everything else, as well as any request Kev is unsure about.

On the 137 held-out requests, picking the model with Kev passed 81.8% of answers, against 83.2% for the best single model, Qwen3.6. That is two requests apart, at 25% less cost, and each answer took 33 seconds on average instead of 51.

The ceiling is higher. Choosing the best model for each request after the fact would pass 94.2% of answers, so a table by kind of work leaves quality that a finer policy could recover.

Speed and cost

Kev runs on an NVIDIA GB10 in Ontario that also serves our other open models. It holds under 9 GB of memory and handles one request at a time.

A short decision takes about 150 milliseconds, and a 4,000-word request about 1.2 seconds. The datacenter GPU of our previous test was faster on short requests and close on long ones.

The price is CAD 0.0532 per million input tokens, and output is free. That is the price TypeSafe publishes for Jev, less ten percent. A three-question decision on a 500-token request costs less than three thousandths of a cent.

Kev bills every token it reads: the state once, and each question with its options. TypeSafe does not publish how Jev counts its tokens, so the two prices are close but not proven identical.

Privacy

Kev keeps nothing. Each request is read once and forgotten with its answer, and nothing is shared between customers, not even a cache whose timing could reveal what someone else sent.

Requests are served in Canada. On a key that requires our PII filter, personal information in the state is pseudonymized before Kev sees it.

What this does not prove

• This is one run on 328 requests that we wrote and labelled ourselves, and the figures come from the 137 we kept aside. In a kind with 10 held-out requests, each mistake moves the figure by ten points.
• The judge is also one of the four models it grades, which may flatter Qwen3.6 on prose. We checked the judge on answers we knew to be right or wrong, not on subtle mistakes.
• The selection was simulated from Kev's answers and our routing table. The gateway's own auto still has to pass the same test before it picks anything for you.
• The latency figures come from a client on the same machine. A request through our API from elsewhere in Canada adds the network time.

What comes next

Kev is the first half of auto, the mode where a request names no model and CARouter picks one at the price or speed you prefer. Before auto picks anything on your behalf, we want a finer routing policy that closes more of the gap to 94%, better detection of long answers, and our own auto passing this test.

In the meantime, Kev is yours to call directly, for support triage, lead qualification, routing or any decision you would rather not leave to a chat model's prose. Every measurement behind this article is kept, and we will share it with anyone who asks.

Today every request to CARouter names the model that answers it. We want to offer one that names none: write auto, and CARouter picks the model best suited to the request, then serves it the way you prefer, cheapest, fastest or balanced.

That takes a decision model, a small model whose only job is to read a request and say what kind of work it is, in a few milliseconds. We tested three candidates on 1,020 real requests, in English and in French, to find out whether one is good enough to trust with that call and what it would cost to run.

The short version

• Accuracy. Jev, a hosted commercial model, identifies the kind of request correctly 81% of the time in English and 76% in French. Kev-4B, an open model we can run ourselves, reaches 76% and 70%. Its smaller sibling Kev-0.8B reaches 70% and 64%. Laya, the third candidate, stays under 60% out of the box.
• Confident mistakes. Jev is wrong while claiming to be sure on one request in eight. Kev-4B is on one in thirty, and Kev-0.8B never was. For a system that should only act when it is sure, that matters more than the accuracy gap.
• Bigger is not better. The largest Kev, at 9 billion parameters, scores no better than the 4-billion one and needs twice the memory. The 4B is the sweet spot.
• Speed. On a datacenter GPU, one decision takes 43 milliseconds on Kev-0.8B and 58 on Kev-4B, an overhead a customer will not notice. Without a GPU, only Laya answers fast enough to be usable, and it is the least accurate.
• The best candidate. Kev-4B: accurate enough, honest about its doubts, open, and fast on a GPU. Whether it is good enough to select models for our customers is a question this test cannot settle, because no public dataset contains the kinds of requests they send. That is the next test to build.

Why pick the model for you

CARouter gives access to a catalog of open models, all served from Canada, through one API. Most customers choose one model when they integrate and never revisit the choice. But a request to fix a piece of code, a request to translate a paragraph and a request to summarize a contract do not need the same model, and the best one for each is rarely the most expensive.

Today a customer can already tell us how to serve a given model: at the lowest price, at the highest speed, or balanced between the two. That preference chooses among the providers that serve the model the customer named. It cannot choose the model itself, and it can never move a request outside Canada. Automatic selection would add the first of those two things while keeping the second.

What a decision model is

Ask a chat model to classify a support ticket and it writes a sentence back, which then has to be read and interpreted. A decision model does not write. It takes a text and a typed question, which of these categories, how much on this scale, yes or no, and returns the answer with a confidence score, in one pass.

There is nothing to interpret and nothing it can make up, because it never generates. That makes it the right tool to sit in front of every request: fast, predictable, and honest about how sure it is.

Jev is the commercial version of this idea, sold by TypeSafe as a hosted service. Kev is an open reproduction of it, in three sizes, that anyone can run on their own hardware. Laya is a second open model that answers the same kinds of questions with a different design.

How we tested

We asked each model one question about 1,020 requests: what kind of task is the user asking for, among nine kinds such as writing code, solving a math problem, summarizing a passage or brainstorming ideas. The requests and their correct answers come from three public datasets that people wrote and labelled by hand, not from us.

The same requests exist in English and in French, so the two languages are compared on identical material. Neither Kev nor Laya had ever seen these datasets. Jev's training data is not public, so it may have.

Speed was measured separately, on three machines: a laptop's processor alone, the same laptop's small GPU, and a rented datacenter GPU. Every request was new, so no shortcut or cache could help.

What we found: accuracy

Jev leads, Kev follows five to six points behind, Laya trails. Two things are true for every model. Code and math are easy: Jev and Kev-4B get them right on every English request, and Kev-4B keeps a perfect score on math in French. And French costs everyone five to eight points, Jev included, so it is a property of the task rather than of one model.

The scores count an answer as right when it lands in the right family of tasks. The datasets themselves label the same kind of request in more than one way, a question about a passage being called a summary in one place and a reading question in another, so the finer nine-way score is lower for every model and in the same order.

The second chart is the one that decides. A system that picks a model on a customer's behalf should only do so when it is sure, and otherwise fall back to a sensible default. That means setting a confidence bar. A wrong answer given with high confidence sails over that bar, so what matters is not only how often a model is wrong, but how often it is wrong while sure.

Jev is the most accurate and the most overconfident: 12% of its answers are confident mistakes. Kev-4B is at 3% and Kev-0.8B at 0%. A confidence bar catches Kev's mistakes and lets Jev's through.

What we found: speed, and where each one can run

On a datacenter GPU every model is fast enough. Kev-0.8B answers in about 45 milliseconds whatever the request length up to a thousand words. Kev-4B needs 58 milliseconds on a short request and about a second on a very long one. A laptop GPU gives the same answers two to seven times more slowly, and cannot hold Kev-4B at all.

Without a GPU, the picture changes. Kev-0.8B takes over a second per decision on a laptop processor, and Kev-4B close to a minute. Only Laya answers in a quarter of a second, about four decisions a second per machine, and it is the least accurate of the six.

Memory decides the rest. Kev-0.8B needs 3.4 GB of GPU memory, Kev-4B 17 GB and Kev-9B 32 GB. On a machine that also serves a language model, that memory is taken from the customers' own requests.

What deploying each one would mean

• As a decision service, on a GPU with room to spare: Kev-4B. Five to six points behind Jev, a quarter as often confidently wrong, free to run under an open licence, and compatible with Jev's API, so a team already using Jev could switch by changing an address. 17 GB of GPU memory and 58 milliseconds would be the price, and the memory is the real cost on a machine that also serves language models.
• As a decision service where memory is tight: Kev-0.8B. 3.4 GB, 45 milliseconds, never confidently wrong in our test. It would give up six points of accuracy against the 4B, and that is the whole trade.
• Behind automatic model selection: Kev, under a confidence bar, once one more test is passed. The 45 milliseconds it would add before an answer starts are acceptable, and its mistakes are the kind a confidence bar catches. What our test does not show is accuracy on the kinds of requests our customers actually send, multi-step agent work and very long documents among them, because no public dataset contains them. That test has to exist, and be passed, before anyone's requests are routed on Kev's word.
• Running without a GPU: Laya, once trained for the job. It is the only candidate that answers quickly on an ordinary processor. Out of the box it reads the task right barely half the time, and its authors say its accuracy comes from training it on your own examples, which we have not done.
• Jev: the reference, not a candidate. It is the most accurate model in the test and a hosted service from a San Francisco company that makes no statement about where requests are processed, which settles it for a product whose whole promise is that requests stay in Canada. It is also the most overconfident, the one flaw a confidence bar cannot fix.

Why the choice has to stay put

A conversation is a series of requests, each one carrying everything said before it. Most providers keep a short-term memory of that shared beginning, so that the next turn is cheaper and faster if it reaches the same provider. CARouter already keeps a conversation with the provider that holds that memory, for as long as it lasts.

If the model chosen for a conversation changed from one turn to the next, that saving would be lost every time, and the tone of the answers would change mid-conversation. So automatic selection has to hold its choice: it remembers the model it picked for a conversation and only changes it when the decision model disagrees with confidence.

That is why the confident-mistake rate is the number this whole test turns on. One confidence bar decides two things, whether a choice is trusted at all and whether a later turn may change it. And if the decision model is ever unavailable, requests go to a default model rather than being refused.

What this does not prove

One question, one run, one set of requests. A differently worded question or a different set could reorder the middle of the table. It could not bring Laya into contention or move Jev off the top.

The processor figures come from a laptop, and a server processor would do better. They also lack the optimized code Kev was written for, which only exists for GPUs. So they are a ceiling on how slow Kev is without a GPU, not a prediction for a given server.

Jev was timed from Canada over the public internet, at 185 milliseconds per decision. That is what a customer here would experience, not the model's own speed.

Where this leaves us

Kev-4B is the candidate we would deploy, and the two open questions are practical ones: which machine can spare 17 GB for it, and whether it holds up on a test built from the requests our customers really send. A decision service customers call directly, the way they call embeddings or reranking today, needs only the first answered; automatic model selection needs both. Every measurement behind this article is kept, and we will share it with anyone who asks.

Our first experiment generated text through encrypted linear operations, but missed our interactive target by a wide margin. This follow-up asks whether current CKKS GPU libraries provide better building blocks for private AI.

Read the first homomorphic inference benchmark.

On an A100 80 GB SXM, a packed 512 → 2,048 projection took 23.43 ms with FIDESlib and 9.89 ms with Cheddar. Batching also increased aggregate throughput. These are validated matrix measurements: we did not generate a complete LLM answer with CKKS, and neither setup demonstrated confidentiality from its host.

Why test CKKS?

CKKS is the Cheon–Kim–Kim–Song scheme for approximate arithmetic on encrypted numbers. It can pack values into one ciphertext and operate on them together. That makes it relevant to the matrix operations inside neural networks, while introducing numerical error that must be measured. The original CKKS paper.

We built pinned revisions of FIDESlib and Cheddar. FIDESlib offers GPU CKKS operations with OpenFHE interoperability. Cheddar provides a GPU implementation with optimized linear transforms. This changes the arithmetic engine from our first experiment; it is not an encryption setting that can be enabled in vLLM. FIDESlib. Cheddar.

What we actually measured

The screen used dimensions from Qwen3.6-35B-A3B, with public synthetic vectors and public synthetic weights. Input values were sampled uniformly from −1 to 1; weights used the same range divided by the square root of the input width. We measured encrypted-vector × plaintext-matrix products, with prepared weights reused across trials.

Every configuration had 1 warm-up followed by 3 measured trials. We synchronized the GPU around each evaluation and compared every output against a plaintext reference, with a fixed maximum absolute error of 0.0001. The figures below show medians; the matrix chart and table also show the observed minimum and maximum, not confidence intervals.

GPU evaluation excludes key generation, weight preparation, encoding, encryption, decryption and network transport. Both cryptographic clients ran on the GPU VM. The separate CPU controller managed collection and deletion; it was not a separate trusted cryptographic client. These placements were suitable for public benchmark inputs, not private customer prompts.

The FIDES matrix configuration enforced OpenFHE’s HEStd_128_classic parameter check, with ring dimension 16,384, depth 2 and 40-bit scaling moduli. Cheddar used its upstream bootparam_40 parameters at level 2, with ring dimension 65,536 and 32-bit arithmetic. The parameter sets and evaluation paths differ, so this compares our configurations rather than isolating a library-only speedup.

Packing matters as much as the library

We first padded rectangular matrices to square layouts. A denser layout reduced unused work by packing diagonals and combining partial sums. In FIDES, the 512 → 2,048 projection fell from 83.13 ms to 23.43 ms. The table uses this denser layout for both libraries. Cheddar uses the alternate minimum-key-switch path that passed our numerical checks.

We also fetched one real gate-projection matrix from the pinned checkpoint, without downloading the full model. FIDES evaluated that 2,048 → 512 projection in 23.74 ms, with maximum absolute error below 0.000001 in the measured trials. The activations remained synthetic. This checks one real-weight operation, not a transformer block, generated text or model quality.

Batch throughput is not chat speed

For Cheddar’s 2,048 → 1,024 projection, packing 16 independent vectors into one ciphertext kept GPU evaluation near 14.4 ms. GPU-only throughput rose from 69 to 1,111 vectors/s. All vectors used the same public matrix and encryption context; this was not a test of batching unrelated customers with different keys.

Including encoding, encryption, decryption and decoding, the measured batch-16 component took 63.23 ms and delivered 253 vectors/s. Preparing its reusable weights took 52.3 seconds and is excluded from those timings. We have not demonstrated that every prepared weight in the full model can remain cached, or measured network and queueing costs.

The rejected trials matter

Our first Cheddar fused matrix-transform configuration returned incorrect results, despite accurate encryption/decryption and a correct single rotation. A non-BSGS configuration and the alternate minimum-key-switch path passed the small check. We used the latter for the reported screen. We did not establish the root cause of the rejected path or claim that every upstream configuration fails.

We also tested FIDES bootstrapping, which refreshes a ciphertext to allow further computation. The 40-bit configuration failed decryption. Higher-precision configurations returned maximum errors of about 0.000207 and 0.000127, both above our fixed 0.0001 gate. These trials stopped during warm-up; we do not publish their durations as validated bootstrap performance.

Cheddar’s supplied client explicitly warns that it is for testing and is not secure. It keeps test secrets on the GPU. FIDES’s parameter check is useful, but our co-located client still exposed its private key to the VM owner. Passing a numerical test does not validate a deployment’s privacy. Cheddar’s client warning.

What this means for the 10 tokens/s target

Our target allows 100 ms per generated token. For scale only, serially summing the measured Cheddar medians for the shared-expert gate, up and down projections across 40 model layers gives 1.2 seconds of GPU work. This illustrative sum excludes routed experts, attention, nonlinear operations, conversation state, the vocabulary projection and client costs. It is not measured model throughput or a universal lower bound: fusion, batching and a different implementation could change it.

We therefore stopped at the component screen instead of claiming a full CKKS inference benchmark. There is no CKKS tokens/s or cost-per-million-tokens result from this experiment. For a batched projection alone, GPU evaluation cost per million vectors is 1,000,000 × P × t / (3,600 × B), where P is the GPU hourly price, t is evaluation time in seconds and B is batch size. Setup, client work, transport and the rest of the model are additional.

Sovereignty and spare GPUs: a research path, not a service yet

Computing without giving a GPU operator readable prompts could eventually widen the pool of usable hardware across the world. This experiment shows that current CKKS libraries merit further work on that path. It does not establish a practical solution to GPU shortages or a complete sovereignty guarantee. More cryptographic computation can also consume more GPU capacity per useful answer.

A usable system still needs a trusted client and key boundary, validated encrypted attention and state, workable nonlinear approximations, acceptable answer quality, and a policy for leakage through expert selection and traffic patterns. Confidentiality alone also cannot force a host to return a correct answer, keep serving or erase its copies. Terraform and dstack can manage machines; these properties require a designed and tested inference protocol.

Further homomorphic work would require a complete validated encrypted block with a separate trusted client, followed by model-quality and end-to-end latency tests if it passes. We are deferring that work. The present evidence does not justify launching encrypted community-GPU chat at our interactive target.

A possible follow-up: confidential VMs with protected GPUs

Could an encrypted virtual machine make community GPU inference practical? The distinction is what the encryption protects. Encrypted disks protect stored files, while an ordinary VM or an added type-2 hypervisor still leaves live memory accessible to the outer host. A hardware-backed confidential VM can protect CPU memory, but ordinary GPU passthrough does not extend that protection to GPU memory and computation.

The architecture worth evaluating combines a confidential CPU VM, a GPU running in a supported confidential-computing mode, and a protected connection between them. An independent verifier would check fresh CPU and GPU attestation evidence, the approved guest workload and its session identity before releasing keys or sending private prompts. GPU passthrough is part of that setup; it is not the security guarantee by itself. Confidential VM and GPU reference architecture.

Unlike homomorphic evaluation, this would let a conventional engine such as vLLM compute on plaintext inside the protected environment. It avoids expanding model operations into homomorphic arithmetic, but its performance on our workload remains to be measured. Trust moves to the hardware security boundary, firmware, approved guest software and attestation and key-release services.

Compatible community hosts could therefore contribute without receiving readable prompts through their normal host administration tools. This does not make arbitrary hardware suitable: the platform must support and expose the complete protected path. The owner can still interrupt service or retain ciphertext copies, and the guarantee does not cover every physical attack or side channel. Confidentiality also does not establish geographic residency or sovereignty by itself. Confidential-computing threat model and limits.

This is a proposed follow-up, deferred for now. We have not run this confidential-VM experiment and report no throughput or security-validation result for it. If we revisit it, we would first verify attestation and refusal of unsupported configurations, then compare the same model and workload with and without confidential mode, measuring first-token latency, decode speed and cost.

Denvr’s compute credits supported this work. The measurements and conclusions are ours. The worker and controller were deleted after the results and source archives were retrieved and verified; the downloaded Qwen matrix and compiled libraries were not retained.

Contact us to request the benchmark data and source files.

Could CARouter use GPUs operated by people we do not trust, while keeping customer prompts confidential? Homomorphic encryption makes that an appealing research question. We started with GPT-2 on an H100, then tested Qwen/Qwen3.6-35B-A3B on a Denvr A100 with 80 GB of GPU memory. The second experiment addresses a much more realistic model for an AI service.

Our interactive requirement was at least 10 output tokens per second per request. The validated 16-token encrypted-linear continuation reached 0.00598 tokens/s, with 35.5 minutes before the first token: about 1,673 times below the throughput target. A token is a small piece of text, often part of a word. We would not build an interactive private-chat service around this implementation.

Thank you to Denvr: its AI Ascend program granted CARouter $1,000 in compute credits, which made these experiments possible. The credits supported the modern-model GPU runs and the dedicated CPU controller. This is an acknowledgement of infrastructure support; the measurements and conclusions are ours.

Explore Denvr AI Ascend.

The privacy problem starts in memory

Encryption protects a connection or a disk, but ordinary inference still handles readable prompts and conversation state in working memory. A conventional host administrator may inspect that memory, including the KV cache. Encrypting a volume during provisioning does not prevent this, and deleting a VM does not prove that its owner erased every copy.

Homomorphic encryption allows a worker to compute on encrypted values and return an encrypted result. Fully homomorphic encryption, or FHE, extends this idea to general computation. It requires a compatible computation engine: Terraform and dstack can manage VM lifecycles, but wrapping vLLM in an encrypted container does not make its arithmetic homomorphic.

From a small-model probe to a modern checkpoint

The initial H100 experiment used GPT-2 to establish that we could generate text through encrypted linear operations and check the results. Its best validated single-request variant, with prepared weights cached, reached 0.16 tokens/s and about 33.2 seconds to the first token. Those short warm runs were useful for testing the machinery, but GPT-2 does not represent the model quality customers expect today.

For the Denvr experiment we pinned the public BF16 checkpoint, tokenizer and software versions. We used PyTorch 2.9.1+cu128, Transformers 5.17.0 and Zama’s Concrete ML Extensions 0.1.9. The Zama library provided the encrypted linear operations; the remaining model computation ran on the client. We first checked that the real checkpoint loaded without missing or mismatched text-model parameters. Tiny random-model tests checked integration only; they contributed no target-model performance figures.

The model uses a mixture of experts: each token activates only selected expert networks. We intercepted functional linear products as well as ordinary linear modules, so routed expert projections and the output vocabulary projection were included. Attention, nonlinear operations, recurrent state, embeddings and sampling remained plaintext client work. This is client-assisted encrypted-linear inference, not a fully encrypted transformer.

How we measured it

We established a native vLLM serving baseline and a native same-framework generation baseline, then a clear quantized reference with the same arithmetic as the encrypted adapter. The adapter quantizes activations and weights to signed 8-bit values; the BF16 checkpoint label does not mean encrypted BF16 arithmetic. The serving baseline used greedy generation, thinking disabled and eager execution, with repeated short and long prompts and a concurrent group. Prefix reuse was possible; this was not maximum-capacity tuning.

For encrypted generation, public synthetic prompts contained 21 input tokens after chat formatting. The longer prompt asked why encryption matters; the shorter asked what two plus two equals. We measured time to the first token, subsequent decode time and whole-request duration separately. Decode tokens/s excludes the first token. Model loading, key generation, public-key upload and VM startup are outside those request timings.

The first placement put client and evaluator on the same GPU VM to measure computation. Its owner could read the client key and plaintext memory. We then ran a separate evaluator and a CPU client limited to 4 CPUs and 12 GiB of memory. The private key stayed in CPU-process memory; the GPU received a public compression key and encrypted activations. Attention and KV state stayed on the CPU VM.

Measured modern-model results

The longer encrypted continuation completed and all its saved output scores matched the quantized reference exactly. It stopped at its output-length limit, partway through a paragraph; this was not a complete-answer quality evaluation. We deliberately stopped the additional repeat after independently validating the completed request. It contributes no extra throughput sample.

The separate-client trial returned “4” followed by the end-of-sequence token. Both count in its two-token output, leaving just one decode step after prefill. That step took 178.5 seconds. This is a network smoke measurement, not a long-conversation throughput estimate. All its saved scores also matched the reference exactly.

The native serving baseline reached about 17.06 tokens/s per request for short prompts. This establishes that the checkpoint is serviceable on the rented hardware; it is not an equal-arithmetic speed comparison with the quantized adapter. Neither native concurrent throughput nor synthetic matrix batching establishes encrypted model batch throughput, which we did not measure on this model.

The native rates are estimates from streamed content and reported token usage; its first-token column is time to first streamed content. The encrypted paths time each model forward pass and token selection. We retain the timing conventions and raw serving samples in our benchmark records.

Correctness came before speed

Our earlier precision investigation found settings that generated text but failed the numerical comparison. The modern-model runs inherited the accepted higher ciphertext-storage and key-switching precision. We checked encrypted integer products in the full-model audit, then saved and independently reloaded every completed encrypted request’s output-score tensors. The comparison tolerance remained 0.005; the observed maximum difference was zero.

This establishes agreement with the tested quantized reference. It does not establish equal answer quality to the original checkpoint: the native and quantized continuations differed. We did not run a task-quality suite or an independent cryptographic security review. Failed setup attempts, a serialization failure and deliberate interruptions remain in the experiment record rather than being counted as successful inference.

The network was not the main delay

The separate-client decode step spent 167.5 seconds inside the evaluator’s request timer. The difference between client round-trip time and that timer was 2.2 seconds, combining transport, serialization and scheduling. It is not pure network latency. Encrypted evaluation and weight handling dominated the measured delay; changing the VM orchestrator would not remove that work.

The network trial transferred 53.3 MB of application payload during that decode step, across 1,031 sequential linear calls. Initial public compression-key upload added 536.9 MB and took 2.09 seconds, separately from inference. These sizes exclude HTTP, SSH and TCP overhead. Comparing total time with the co-located smoke does not isolate networking either: client execution and integer-audit instrumentation also differed.

We separately tested caching prepared weights on the GPU. It improved some single-row component timings, but that microbenchmark was not a cached full-model generation run. An isolated kernel speedup cannot be presented as a new model throughput result.

What an answer costs

At the recorded GPU quote of $1.45/hour, the longer continuation consumed about $1.87 of measured request runtime. Its steady decode rate projects to roughly $67,394 per million output tokens, excluding prefill and setup. With the separate CPU allocation included, the short network request consumed about $1.24 of runtime. These are price-dependent calculations, not invoices or million-token capacity tests.

Estimated compute for the Denvr experiment, including failed attempts, provisioning and the controller’s lifetime, was $7.75. That estimate comes from allocation windows and hourly quotes; storage, networking, billing rounding and taxes are outside it. We did not consume the entire grant. The credits let us answer the engineering question before committing to a larger system.

What this means for private CARouter inference

For the tested engine and configuration, the interactive performance gate failed. That is evidence against building production orchestration around this implementation today, not proof that every homomorphic scheme or future optimization must be slow. A different engine must demonstrate useful speed, acceptable answer quality and its complete privacy boundary on the target workload.

The separate client also leaves explicit trust assumptions. Selected expert identifiers, tensor shapes and timing are visible to the evaluator. The ordinary CPU VM assumes its own infrastructure operator is trusted. Our synthetic prompts were public source fixtures, so the test does not demonstrate that their text was unknown to the GPU host. It does not make arbitrary community machines secure for customer prompts.

For native inference, a separate direction is confidential computing with compatible CPU and GPU protection, attestation and key release bound to the verified workload. We did not deploy or benchmark that architecture here. A GPU model name alone does not establish those protections. Confidentiality also does not attest a machine’s location or replace CARouter’s residency and provider-consent rules.

An independent CPU controller owned allocation deadlines and collected results, so a laptop disconnection did not control GPU billing. We retrieved the source and evidence, verified their hashes and confirmed through Denvr’s API that every experiment VM was deleted, including the credential-bearing controller. We retained output tensors for follow-up checks, not model weights or compiled binaries. API-confirmed deletion is an operational observation, not proof of physical erasure or protection against a host that previously saw plaintext.

Where the technology stands today

As of September 2026, research systems are tackling encrypted KV caches and hybrid execution combining homomorphic encryption with confidential computing. These are concrete paths forward, but results depend on model size, trust assumptions and whether a figure measures actual generation or projects it from components. We have not reproduced those systems, and their headline speeds cannot validate our workload.

Cachemir · Bifrost

For worldwide sovereignty concerns, encryption can reduce dependence on a remote operator’s ability or willingness to protect plaintext. It does not by itself establish sovereignty. Control of keys and the trusted client, permitted compute locations, metadata exposure, model integrity and the ability to keep serving remain separate requirements. A confidential calculation abroad does not automatically satisfy a customer’s residency policy.

For GPU scarcity, the relevant measure is useful answers per unit of compute. Encryption could make a larger pool of otherwise unsuitable machines usable, but its overhead must be smaller than the capacity that access unlocks. Our tested engine spends much more compute time per answer. Adding machines to this unchanged serial implementation would not demonstrate a faster individual conversation.

Our assessment is that the cryptographic building blocks are real, while a general-purpose, economically useful encrypted modern-LLM service on arbitrary community GPUs remains unproven by the evidence we evaluated. Today, our practical direction is trusted compute and separately evaluated attested confidential-computing deployments. We would revisit homomorphic community inference when a complete system demonstrates the required per-request speed, answer quality, protected state and metadata, and recoverable operation. That would address part of the sovereignty problem; it would still need to earn its place as a response to GPU shortages.

Request the benchmark files

These findings describe our September 2026 experiments. We retain the detailed measurements and source files for follow-up work. The earlier H100 experiment used a different model and hardware, so the results do not compare GPU generations.

Contact us to request the benchmark data and source files.

Further reading

• Concrete ML client-assisted inference documentation.
• Encrypted CUDA extension source.
• NVIDIA confidential-computing attestation prerequisites.

An AI assistant often receives the same instructions and conversation history with each new question. Processing that repeated material takes time and can add to the bill. Prompt caching lets a provider temporarily reuse work it has already done, which can mean faster responses and lower input costs.

CARouter’s cache affinity is designed to help retain that benefit when a request has to move to another provider. It gives the provider that last answered a preference for subsequent requests using the same API key and model. Activation depends on verifying each provider’s published cache lifetime.

Why a provider change matters

Imagine a customer service assistant handling a conversation. Its usual provider is busy, so CARouter sends the next request to another approved provider running the same model. That provider processes the conversation and may cache the repeated material for later use.

Without cache affinity, the next request can return to the original provider as soon as it is available. The work just done at the replacement provider cannot be reused there. Staying with the provider that just answered gives the next request a chance to benefit from its cache.

Your routing safeguards still come first

• Residency and reliability: the preference cannot move a foreign provider ahead of a sovereign one, override your provider permissions or border controls, or put a degraded provider ahead of a healthy one in the same residency tier.
• Cost control: if your API key prioritizes cost, the lowest listed price still comes first. Cache affinity only breaks a tie between equally priced offers.
• Limited routing memory: this feature remembers which provider last answered for an API key and model. It does not store, copy or fingerprint your prompt. The provider manages its own cache.
• Temporary preference: each successful request renews the preference for the provider’s verified cache lifetime. Once it expires, normal routing resumes. If the preference cannot be read, normal routing also applies.

What to expect

The opportunity is greatest for repeated instructions and ongoing conversations. Savings and faster responses are possible, not guaranteed: the provider must support caching, recognize reusable content and still have it available. CARouter does not create a cache or transfer one between providers.

Where a provider reports cached input usage and has a published cached rate, that rate is used in the request charge. The documentation shows cached input prices where available, so your team can assess the potential benefit.

Cache affinity is enabled only for providers whose cache lifetime has been verified. Until a provider has that verification, requests follow the existing routing rules.

Compare available models and prices. Read the technical routing rules.

Every offer on our documentation page now carries a small leaf. Beside it is a number: an estimate of the carbon emitted to serve a million tokens from that model, at that provider. This post says why we publish it, how it is computed, and what it leaves out.

Why a number, and why now

How much electricity AI uses has become a public question, and a fair one. The people who choose CARouter are already making a deliberate choice about where their data goes. We think they deserve the same clarity about what their requests cost the grid, so that sovereign and sustainable can be one decision instead of two.

On 3 September the Government of Canada launched its Responsible Data Centre Development Principles. Two of the five ask data centres to minimize environmental impacts and to be transparent about them. We do not build data centres. We route requests to them. Transparency at our level means a figure beside each offer, with its method and sources, so that a customer can act on it. The principles, on canada.ca.

What the leaf says

• The figure. Grams of CO₂ equivalent per million tokens, operational electricity only, rounded to two significant figures.
• The colour. The serving grid's band: green below 100 g CO₂e per kWh, amber up to 300, red above. The grid, not the figure, because the grid is what you can act on by choosing where a model is served from.
• Per offer, not per model. The same model served from Quebec and from Alberta differs by two orders of magnitude. A per-model average would describe a grid nobody serves from.
• Or nothing. An offer whose model publishes no parameter count, or whose region has no sourced grid figure, shows no leaf. We do not fill a gap with a neighbour's number.

How it is computed

Two verified inputs and one published formula.

Energy comes from the model's active parameter count, the one its maker publishes on the model card. A forward pass costs two floating-point operations per active parameter per token, for prefill, decode, embeddings and rerank alike. On the H100-class serving constants Epoch AI published, that is about 0.006 kWh per billion active parameters per million tokens, cooling and distribution included.

Carbon intensity comes from the grid of the region the provider claims to serve from: the Canada Energy Regulator's figure for each province, RTE's for France, the Energy Information Administration's for the United States. Each row carries the year it describes and the URL it was read from. A provider that claims only Canada, without a province, is costed on the national average.

Multiply the two. For a model with 100 billion active parameters, 0.59 kWh per million tokens becomes, on the published grid figures:

• 0.7 g in Quebec, 8 g in British Columbia, 21 g in Ontario;
• 59 g on the Canadian national average;
• 220 g on the US average, and 280 g in Alberta.

Same model, same tokens, a factor of four hundred between the cleanest and the dirtiest grid. That spread is the whole reason the figure is per offer, and the whole reason it is worth showing.

What it leaves out

• Embodied emissions from manufacturing the hardware, and the network in between. It is electricity for inference, nothing else.
• The provider's actual hardware. Every offer is costed on the same H100 constants, so the figure compares offers with each other and measures nobody's rack. A provider on newer silicon is over-counted, one on older under-counted, by a constant factor.
• Renewable certificates and supply contracts. The grid figure is location-based: a plug in Alberta draws Alberta's mix whatever the paperwork says, and a location-based figure cannot be gamed.
• Freshness. Grid figures are annual averages and drift like a price. The year on each row says how old it is, and re-reading the source is a maintenance duty we have taken on.

Using it to choose

Two habits pay off. Pick the smallest model that does the job, because energy scales with active parameters and a small model on a clean grid is the cheapest token there is. And when a model is served from more than one place, prefer the greener offer. In Canada, that is rarely a trade-off with sovereignty: hydro provinces are both.

The estimate is also in the machine-readable copy of our documentation, with the full methodology, so that an agent choosing a model on your behalf can weigh it too. The documentation page shows the leaf beside every offer that has one.

CARouter has been accepted into Denvr AI Ascend, the startup program of Denvr Dataworks.

Denvr is a sovereign cloud provider from Alberta: Canadian-owned, running its own GPU clusters in Calgary. That is the bar we hold every provider on the gateway to, so the fit is a natural one.

What Denvr is

Denvr passes both questions we ask of a provider: who owns the company, and where the compute sits.

Denvr told us in August what they sell: GPUs by the hour, no inference API. Hosted endpoints are not on their roadmap this year, so Denvr is not among the providers our documentation page lists.

This announcement changes none of it. No request through the gateway runs on Denvr. A startup program is not a partnership agreement. It is credits, and credits buy hours on machines.

What the hours are for

Everything on this blog so far was measured on one desktop box, the DGX Spark-class machine behind CARouter Cloud. It says a lot about small models on unified memory. It says nothing about the H100-class servers Canadian providers actually run.

So the credits go to measurement. Three things:

• Open-weight models on Canadian hardware. The same throughput probe, paired coding evaluation and long-context grid we run on the Spark, on datacenter GPUs in Calgary.
• Numbers the router can use. Today the candidate order is residency, then priority, then price. It knows nothing about how an offer behaves under load. Measured latency and throughput per model is what turns that policy into an informed one.
• Published here, with the mistakes. Each run becomes an article: the protocol, the charts rendered from the same numbers, the caveats. A measurement nobody can reproduce is a marketing claim.

What we are not promising

There is one path by which Denvr hardware could serve customer traffic: CARouter Cloud running its own serving stack on GPUs rented in Calgary. Canadian-owned hardware, a Canadian datacenter, operated by us. Every claim the gateway makes would hold.

Our capacity research already treats a reserved Denvr node as the fallback if the hardware we intend to buy does not work out. That is a possibility, noted for completeness. It is not a plan, and this program does not fund it. Denvr stays off the documentation page until something real changes.

Thank you to the Denvr team for the welcome. The documentation lists every provider on the gateway with the claim each one makes today.

Today we are announcing a partnership with Augure, a Toronto-based AI company. Rosedale One and Tofino 3 are live on the gateway.

The share card carries three chips. Each one is backed by a signed agreement. Here is what they mean in practice.

The short version

• Who. Augure is Canadian-owned, headquartered in Toronto, with no foreign parent anywhere in the chain. It post-trains its own models from open-source bases and serves them on GPUs in Canada.
• What. Rosedale One is the reasoning tier: a million tokens of context, with a thinking phase. Tofino 3 is the light tier: fast, no thinking, 256K of context. Both take tools, structured JSON and images. We measured that ourselves.
• Where. Canada, by contract. A request is served on infrastructure in Canada or refused. Never failed over, mirrored or burst abroad.
• Retention. Prompts and completions are gone once the request is served. Metadata, with no content in it, is kept at most thirty days. We publish those as two separate claims.
• Proof. Every response carries X-CARouter-Region. For these two models the value is read from what Augure reports for that request, not from a label we wrote down once.

Served in Canada, or refused

Augure commits that requests routed through CARouter to these two models run only on infrastructure in Canada. When Canadian capacity runs out, Augure returns an error rather than serving the request elsewhere. A request that quietly leaves the country is the one outcome neither of us accepts, because we have already told the caller it would not.

Our side is the same promise. CARouter never fails over across a border unless the API key has explicitly allowed it. When Augure is full, the caller gets a provider_at_capacity error and a Retry-After header. A less careful gateway would quietly serve another model from another country. We would rather explain an error than a substitution.

Where the region header comes from

Augure's API reports, on every response, where the model actually ran. We read that value for each request and publish it in X-CARouter-Region. Measured from our own credential, both models report Waterloo, Ontario.

So the residency claim is verified on every request, not taken on trust. A value we cannot place resolves to UNVERIFIED, never to the provider's claim.

Streaming writes the header before Augure has said anything. So the corrected value goes out as a trailing carouter.region frame, and into the request log beside the row.

Zero retention, in writing

Augure's security page already said zero retention. What the agreement adds is a commitment that cannot change without notice: prompt and completion content is not kept once a request is served, and none of it is used to train, fine-tune or evaluate any model. Request metadata, with no content in it, is kept at most thirty days, and we publish that as its own claim.

Using it

Augure is off by default in every workspace, and no deploy will switch it on for you. Enabling it in Settings is your acceptance of Augure's own terms. A switch already on would be an acceptance nobody gave.

Once it is on, both models answer at the usual endpoint:

curl https://carouter.ai/v1/chat/completions \
  -H "Authorization: Bearer $CAROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "tofino-3", "messages": [{"role": "user", "content": "Bonjour"}]}' \
  -i | grep -i x-carouter-region

Prices are base rates plus your plan's markup, like every other row. The documentation lists both models with their current rates.

What comes next

More models will come as Augure pins them to Canada. Each one is published only once Augure has confirmed in writing that it is served in Canada, with no failover abroad. A model cannot reach the gateway without that guarantee attached. That is the point of the partnership.

Qwen released Qwen3.8-Flash-Next on 26 August, an early look at the architecture the Qwen4 family will be built on, and it went straight to the top of the coding leaderboards: 91.9 on LiveCodeBench v6 against the 80.4 of the model our own coding endpoint runs. Three days later we had it serving on the same desktop box that serves CARouter Cloud, and we did not keep it. Here is every number we measured, and what they cost.

The previous article: why the coding endpoint runs Qwen3.6 35B-A3B, an older and lower-scoring model, instead of Qwen3.8 27B.

The short version

• It does not fit. The smallest build we can serve is 125.9 GiB. The machine has 121.6 GiB of memory in total. This is the first model we have looked at that is larger than the computer, before a single token of conversation is stored.
• 38% of it is not a model. It is a lookup table: 20,000,000 rows of n-gram embeddings, 51.2B parameters, 47.7 GiB. A token touches 16 rows of it. Stream those from the SSD instead of keeping them in memory and what has to be resident drops to 78.2 GiB, which fits.
• The headline parameter count leaves it out. The model card says 125B. The weights hold 180B. The difference is that table, plus the draft head and the vision tower, all counted separately.
• It is about three times slower. 21.3 tokens a second on 1 stream against 62.2, and 129.8 against 367.2 with 16 streams in flight. Prefill is 3.7x slower. It holds a quarter of the concurrent 262K-token conversations: 1.7 against 7.0.
• It probably codes better. On the 40 problems it finished inside a 32,768-token budget it solved 29, against 27 for the model we serve and 26 for the one before it. It ran past the budget on the other 10. Neither margin is statistically significant.
• And it costs 10 times the wait. It writes 16,751 tokens per problem where the incumbent writes 4,738. At its own decode speed that is a 13-minute answer against 76 seconds, for at best 1 extra problem solved in 40.

A model larger than the computer

CARouter Cloud's self-hosted endpoint runs on one ASUS Ascent GX10, a DGX Spark-class desktop box: a GB10 chip with 128 GB of memory shared between the processor and the graphics side, of which 121.6 GiB is actually addressable. Every model we have shipped had to fit in that, next to the 2 embedding models and the reranker the product also serves.

Qwen3.8-Flash-Next does not. 9 different people had published quantized versions of it within 72 hours, and we checked all of them: the smallest one we could actually serve is 125.9 GiB of weights. That is 4.3 GiB more than the entire machine holds, before the operating system, before the inference engine, before storing one word of anyone's conversation.

Normally that is the end of the story, and our screening process is designed to end it there in about 10 minutes, from published metadata, without downloading anything. What made this one worth continuing was reading the actual list of tensors in the file rather than the model card.

The card says 125B parameters. The weights contain 180B. The gap is one enormous lookup table - the n-gram embeddings, exactly 20,000,000 rows of 2,560 numbers each, 51.2B parameters and 47.7 GiB even after every quantizer in the wild has had a go at it, and not one of them compressed it below 8 bits. It is 38% of the file. Add the draft head and the vision tower, also counted outside the headline, and 125B becomes 180B.

A lookup table is not a matrix multiplication. Producing a token touches 16 of those 20,000,000 rows, about 2.5 KB. So it does not need to be in memory at all: it can sit on the SSD and be read as needed. Somebody had already done exactly this - a public project that patches the official engine to memory-map the table from disk - and with it the resident weights fall to 78.2 GiB and the model runs on one box, alone.

What it took to run at all

6 patches on top of the official image, none of them ours: the memory-mapped table, two fixes for the linear-attention kernels on this particular chip, a guard on a state-copy race, a correction to prefix caching that was silently restoring an all-zero state on every cache hit, and a replacement for a sparse-attention kernel that returns different answers to the same question. The engine itself is not a release - it is a build from a branch, because the architecture is too new to be in any published version, including the one we pin.

It also cannot share the box. Our endpoint normally runs 4 services at once: the coding model, 2 embedding models and a reranker. Qwen3.8-Flash-Next takes the whole machine. Every measurement below was taken with the rest of the product's inference stopped.

The throughput

We ran the same benchmark we run on everything, unchanged: the same prompt, the same 20-second warm-up and 40-second measurement window, the same counters read from the engine itself.

Between 2.8 and 3.4 times slower at every concurrency, 3.7 times slower to read a prompt, and holding 1.7 full-length conversations at once where the model we serve holds 7.0. What makes this worse than it looks is the handicap: Qwen3.8-Flash-Next was running with speculative decoding - a draft head built into its own weights, accepting 51% of its guesses and delivering 2.02 tokens per step - on the fast kernel for this chip. The model that beat it has no draft model available at all and runs on a fallback kernel we have never managed to replace.

The coding score

50 coding problems, the same 50 we score every model on, run identically: same problems, same budget, same settings, one model after the other on the same box.

The first thing the run measures is not quality, it is length. Flash-Next is a reasoning model and it writes 16,751 output tokens per problem, against 4,738 for the model we serve. Our budget was 32,768 tokens, and it ran past that ceiling on 10 of the 50 problems - cut off mid-answer, scoring zero on each. Those are not wrong answers, so the comparable set is the 40 it finished inside the budget.

On those 40 it comes first: 29 solved, against 27 for the model we serve and 26 for the one before it. That is the same order as the public leaderboards, and neither margin is statistically significant - detecting the 11.5-point gap the leaderboards claim would take roughly 88 paired problems. Read it as consistent with Flash-Next being the stronger coder, and too small a sample to prove it.

The overrun rate is a result in its own right, and the one most likely to matter to you: 20% of ordinary coding problems need more than 32,768 tokens from this model. Whatever budget you set, it will hit it more often than the models you are used to, and a truncated answer is worse than a slow one.

What you actually wait for

The verbosity is not an artefact of our harness, though. It is the product.

3.5x the tokens, at a third of the speed, is 10x the wait. Seventy-six seconds becomes 13 minutes, for an answer that is at best 1 problem in 40 better and is usually identical. And a model that overruns a 32,768-token budget on a fifth of ordinary coding problems will do that to a paying caller too, not only to our test harness - a truncated answer is worse than a slow one.

The honest caveat

40 problems cannot settle whether one model is 10% better than another at writing code, and we are not claiming Qwen3.8-Flash-Next is worse. We think it is probably better. What we measured is that on this hardware its advantage is unreachable: it arrives ten times later, in a quarter of the parallel conversations, on an engine built from a branch, with nothing else running on the machine.

A different box changes that answer entirely. 2 of these machines, or one with more memory, and the table stays in memory, the model shares the hardware with nothing, and the comparison is a fair fight it might well win. 3 of the models we have rejected this month were rejected for the same reason, which is not really a fact about the models.

What would change our mind

• A quantized table. Nobody has compressed the n-gram embeddings below 8 bits, including the people who quantized everything else to 4. At 4 bits the file drops by 23.8 GiB and stays in memory.
• A quantized linear-attention path. 43% of what the model reads per token is projections that every published version leaves at full width. Halving those roughly doubles the ceiling.
• The architecture in a released engine. Today it exists on a branch and needs a kernel fix for this chip that upstream has not taken. That turns a patched fork back into a version number.
• A second box. The configuration everyone running this model successfully is using.

Until one of those, the coding endpoint keeps running a smaller, older, lower-scoring model that answers in 76 seconds and holds 7 conversations at once. Everything above is reproducible from our infrastructure repository: the screening with its arithmetic, the benchmark script, and the eval configuration.

For four days CARouter Cloud served Qwen3.8 27B, and the three posts before this one are the story of making it fast: a better draft model, then better settings for that draft, which together took it from 20.8 to 34.8 tokens a second per stream. It was not enough. A dense 27B on a box with 273 GB/s of memory bandwidth has to read all 23.4 GB of itself to produce one token, and no amount of tuning changes that arithmetic. Today the coding endpoint runs Qwen3.6 35B-A3B instead - an older model, one that scores lower on the public coding leaderboards, and a bigger one at 35B against 27B. It is also between one and a half and two times faster at everything we measured, and we could not find the quality we were supposed to be giving up.

The previous article: tuning speculative decoding on the same box, two thirds faster coding answers.

The short version

• The problem. A dense model reads every one of its parameters for every token it writes. On the GB10's 273 GB/s that put a hard ceiling of about 12 tokens a second on Qwen3.8 27B before any speculation, and the practical consequence was queueing: on an automated agentic platform, a routine coding build took an hour.
• The fix, which sounds wrong. A bigger model. Qwen3.6 35B-A3B has 35B parameters but activates only 3B of them per token, so it reads 2.0 GB per forward pass instead of 23.4. It occupies the same 21.8 GiB on disk.
• The throughput. 62.2 tokens a second single-stream against 34.8, and 367 against 241 with sixteen streams in flight - between 1.5x and 1.9x at every concurrency we measured. Prefill is 3.2x: about 6,600 tokens a second against 2,088 on a 12K prompt. The same build that took an hour takes ten minutes.
• The concurrency. From the same 36 GiB of KV budget it holds 1,837,064 tokens against 813,427 - 7.0 full 262K-token sessions at once against 3.1. Only 10 of its 40 layers keep a cache that grows with the conversation; the other 30 keep a constant.
• The quality we expected to lose. The public leaderboards put it about ten points behind on LiveCodeBench v6. On the fifty BigCodeBench problems we run ourselves, paired on the same problems on the same box, it scored 33/50 against the 27B's 32/50 - a difference of one problem, with an exact McNemar p of 1.000. On long-context retrieval it was better: 10/10 against 9/10, and 10/10 against 8/10 at 220K tokens.
• The honest caveat. Fifty problems cannot detect a ten-point gap; that would take about 218. So the claim is not that the two models are equally intelligent - it is that we looked for the difference on our own workload and could not find it, while the speed difference was impossible to miss.

Why a dense model runs out of road

Generating a token means reading the model's weights out of memory and multiplying. For a dense model that is all of the weights, every single time. Qwen3.8 27B in NVFP4 is 23.4 GB of them, and the GB10 moves about 273 GB/s, so the fastest it can possibly write is 273 divided by 23.4 - about 12 tokens a second. Everything the previous two posts achieved was speculative decoding working around that number rather than changing it: a draft model guesses several tokens ahead, the big model checks them in one pass, and the accepted ones come out at once. It got us to 34.8 tokens a second, which is roughly three accepted tokens per 12-per-second step.

A sparse mixture-of-experts model breaks the arithmetic instead of working around it. Qwen3.6 35B-A3B holds 256 experts and routes each token to 8 of them, so although all 35B parameters sit in memory, only about 3B are read per forward pass - 2.0 GB instead of 23.4. The ceiling moves from 12 tokens a second to about 140. The model is larger and the file on disk is the same size; what changed is how much of it has to be touched to produce a word.

What we measured

Both models on the same box, same 36 GiB KV budget, same 16-sequence limit, one swapped for the other because the two do not fit together. Aggregate decode across concurrent streams:

Two details make that chart understate the case. The 35B-A3B ran with no speculative decoding at all, while the 27B had the seven-token DFlash2 draft the last post shipped, landing about 3.9 accepted tokens per step. And its experts ran on vLLM's Marlin kernel, which unpacks the 4-bit weights and multiplies them in 16-bit, where the 27B gets FlashInfer's CUTLASS path. That second one is a property of the checkpoint rather than of the box: NVIDIA quantizes this model W4A16, four-bit weights against sixteen-bit activations, and the CUTLASS kernels want the activations in FP4 too, along with calibration scales a W4A16 checkpoint never carries. So these are floor numbers on both counts.

Memory is the other half, and on a shared-memory box it is the half that decides how many people you can serve at once:

The concurrency figure comes from the engine's own boot log: 1,837,064 cached tokens against 813,427 from an identical budget. The reason is architectural. Only 10 of the 35B-A3B's 40 layers use the kind of attention whose cache grows with the length of the conversation; the other 30 are linear-attention layers that keep a fixed-size state per session no matter how long it runs. The dense 27B keeps a growing cache on 16 of its 64 layers, and each of those costs more per token. Twice the sessions, from the same memory, is not a tuning win - it is what the layer layout gives you.

The part we expected to hurt

Qwen3.6 is an older generation than Qwen3.8, and the published coding benchmarks say so: LiveCodeBench v6 puts the sparse model at 80.4 against the dense model's 90.3. Ten points is not a rounding error, and we did not want to discover it in production.

So we ran the same fifty BigCodeBench problems against both, on the same box, in the same Docker sandbox, uncapped, with identical settings. Then we did something we should have done sooner: we re-measured the 27B rather than trusting its score from four days earlier. That old number had been collected under a different draft, a different vLLM version, and a run that the harness's own timeout bug had broken into pieces and restarted. Re-run cleanly, the dense model scored 32/50 where it had scored 33/50 before.

Paired on the same fifty problems, the two models agree on 43 of them - 29 both solved, 14 both failed. The sparse model alone solves four, the dense model alone solves three, and an exact McNemar test on those seven disagreements returns a two-sided p of 1.000. That is the flattest result a paired test can give.

Long-context retrieval went the other way entirely. We hide a fact in a haystack of up to 220,000 tokens and ask the model to find it, at three depths for each of four lengths. The dense model scores 9 out of 10 overall and drops to 8 at the longest setting. The sparse model scores 10 out of 10, at every length and every depth. That was the result most likely to disqualify it - we sell a 262,144-token context, and a model that answers quickly but loses the thread deep in a long file would be a regression no coding score would reveal - and it is the one that came back strongest.

It also reached the same score using far fewer words: 236,949 output tokens for the fifty problems against 558,275, with a median sample of 4,773 tokens against 9,142. The dense model hit our 32,768-token ceiling on at least one problem; the sparse model never passed 12,253. Both models were reasoning - the sparse one logged 219,951 reasoning tokens across all fifty samples - so this is brevity rather than a thinking mode left switched off. For an endpoint that bills per token and holds a context per session, arriving at the same answer in less than half the words is worth as much as the tokens-per-second.

What we cannot tell you

Fifty problems is not enough to detect a ten-point difference. Detecting an effect that small at 80% power needs roughly 218 paired problems, and running those costs a day of box time during which the endpoint serves nobody. So our result does not refute the leaderboard; it means that on the one coding benchmark we run ourselves, with both models measured the same week on the same hardware, we could not find the gap.

We also measured coding and long-context retrieval, and nothing else. General reasoning, instruction following and multi-turn agentic behaviour are untested here, and the published differences on those may well be real. What we can say is that on an automated agentic platform running long autonomous builds, several basic coding tasks that took an hour on the dense model finish in about ten minutes on this one, with no quality drop we could see in the output.

The honest shape of the decision is this: we traded a difference we could not measure for a difference we could not miss. A request that queues, or streams at half speed, is a worse outcome for the person waiting than a marginally weaker answer that arrives promptly - and a model too slow to serve the demand is worth nothing at all to the requests it turns away.

What is running now

Qwen3.6 35B-A3B in NVFP4, on the same GB10 behind the same TLS proxy, at the same price: $0.20 per million input tokens and $0.80 per million output. It answers to qwen3.6-35b-a3b, keeps the full 262,144-token context, reads images, calls tools and returns its reasoning. Qwen3.8 27B is retired from the catalog, and because it was served only from our own hardware it is simply gone rather than routed elsewhere.

Its configuration file stays in the repository, and switching back is one line. Two things are still open, and we had both of them wrong when this post first went up. The draft model does not need publishing by anyone: this checkpoint already carries a multi-token-prediction head, one attention-and-experts layer trained to guess the token after next, sitting unused in weights we had already downloaded. And the kernel is not a matter of asking for CUTLASS, which no W4A16 checkpoint can use; it means either a build of this model quantized for FP4 activations, or vLLM's fused SM12x kernel, which is written for exactly this chip and simply is not picked automatically. Both are measurements we owe you rather than improvements we can claim, and every number above was measured without either.

Speculative decoding has a small "draft" model guess several tokens ahead so the big one can check them all in a single pass, keeping the ones it agrees with. Which guesses the draft makes, and how they get checked, turn out to be two settings we had left at their defaults. Changing them - two keys in one config file, no new hardware, no change to what the model writes - took the coding endpoint on our self-hosted box from 20.8 to 34.8 tokens a second per stream: two thirds faster than the model's own built-in guesser, and 22% faster than what we shipped a week ago. Here is what each setting does, why each one is a loss on its own, and the conclusion from last week that the pair of them overturned.

The previous article: MTP vs DFlash2 on CARouter's DGX Spark, 35% more tokens per step, not a faster step.

The short version

• What a client sees. The coding endpoint writes an answer two thirds faster than it did before any of this tuning: a token every 29 ms instead of 48, a typical 460-token reply in about 13 seconds instead of 22. Nothing about the answers themselves changes.
• The setting that paid. draft_sample_method: "probabilistic" makes the draft sample from its own distribution at the request's temperature, shaken by the same random noise as the big model's own sampling, instead of always taking its single best guess. The acceptance test then compares two real distributions instead of reading a probability off a certainty.
• Neither setting pays alone. Block verification, the lossless acceptance rule we recommended at the end of the last article, loses on its own: 22% down at two streams and 11% down at eight, at the temperature clients get by default. Probabilistic drafting alone wins from two streams up and gives up a tenth of single-stream decode, which is exactly where a coding agent sits. Together they cover each other.
• And we had been wrong in an interesting way. Last week we measured the drafter's deepest guesses, found they did not pay for themselves, and shipped a shortened block. That was true of the default setting, which we had not noticed was a premise at all.
• So the depth answer flipped. With the draft sampled probabilistically we re-ran the block-length sweep over four, five, six and seven. The drafter's native seven-token block won: 34.8 tokens/s single-stream against the five-token block's 29.3, and 28.6 against 24.7 at the temperature clients get by default.
• The cost. Nothing measurable. The draft's cached distribution is about 106 MiB, absorbed; the weight load is unchanged at 24.92 GiB; the KV budget gives up 3% to the deeper block's own cache; prompt processing does not move.

What we said last week, and why it was conditional

Last week's sweep found that positions four to seven of the drafter's native block were accepted 28%, 22%, 15% and 12% of the time. Those four positions added 0.77 accepted tokens per step, which is free at one or two streams and real work at sixteen, so shortening the block to five kept the single-stream gain and turned the batched regime positive at every size. We wrote "a static five does the job on this box" and shipped it.

Read that again with a week's hindsight. Every number in it is an acceptance number, and acceptance is not a property of the box, or of the model, or of the workload alone. It is a property of how the draft is produced and how it is checked. We measured one drafting rule - take the drafter's best guess, every time - and drew a conclusion about block depth as though depth and drafting were independent. They are not, and this is the part of the whole exercise that generalises: whenever the drafting rule changes, the depth question has to be asked again.

The setting: sampling the draft the way the model samples

vLLM's speculative configuration has a key we had dismissed. draft_sample_method defaults to "greedy": the drafter proposes its single most likely continuation, and the acceptance test treats that proposal as a certainty - probability one on the chosen token, zero everywhere else. Set it to "probabilistic" and two things change.

The first is in the drafter. DFlash2's selector walks a beam of sixteen candidates per position; in the default mode it takes the highest-scoring one, and in probabilistic mode it samples among them at the request's own temperature. The kernel that does this carries a comment worth quoting, because it is the whole idea in one line: "Candidate ids key the noise, matching the target's own sampling." The draft and the big model are shaken by the same random noise, so when the big model samples stochastically the draft tends to make the same stochastic choice.

The second is in the checker. With a real distribution behind it, the drafter's scores are cached and the acceptance test becomes the true ratio of the big model's probability to the draft's, instead of a probability read off a certainty. That is what makes the second setting - block verification - mean anything at all.

We had written in our own notes that this option applied to Qwen's built-in head and not to this drafter. That was wrong, and reading the code is what corrected it: the DFlash2 speculator overrides the exact method that allocates the cache this mode needs, specifically so that it works with it.

Finding 1: the lossless setting we recommended, alone, is a loss

Last week's closing line named block verification as the next lever: a lossless acceptance rule that checks the whole drafted block jointly rather than one token at a time. It is genuinely there, it works with this drafter, and it costs no memory - the boot prints the same weight load and the same cache size. It also loses. At temperature 0.7 it lands inside the existing spread everywhere, which is what the theory predicts: block verification is provably identical to the ordinary rule when nothing is sampled at random, and a code probe at 0.7 is close to that. At temperature 1.0, where the gain was supposed to be, it gives 37.8 tokens/s at two streams against 48.7 and 120.1 at eight against 134.7 - and that eight-stream figure is three runs within 2.2 tokens/s of each other, so it is not noise. Acceptance never rose.

The reason is the one the previous section sets up. A best-guess draft is a certainty, so the joint rule has almost nothing to work with that the token-by-token rule was not already getting, while its extra arithmetic is charged on every step and bites once the batch is wide. A lossless rule is not automatically a free one.

Finding 2: probabilistic drafting wins at batch and loses where the agent sits

Sampled probabilistically at the five-token block, decode is 25.7, 57.1, 101.2, 163.5 and 246.0 tokens/s at one, two, four, eight and sixteen streams, against 28.5, 47.5, 91.4, 137.4 and 216.1 for the same block with the default setting. From two streams up that is +20%, +11%, +19% and +14%, and acceptance rises with it - 3.0 to 3.9 tokens per step where the best-guess draft managed 2.7 to 3.0. At one stream it gives up a tenth, and one stream is where a coding agent actually sits.

At the temperature clients get by default the picture inverts: single-stream goes to 26.0 against 21.0, +24%, because coupling the draft's randomness to the big model's matters most exactly where the big model is most random. A setting worth +24% or -10% single-stream depending on a parameter the client controls is not a setting you ship on its own.

Finding 3: together, and then the depth answer flips

Adding block verification on top of probabilistic drafting repairs the single-stream cell - 29.3 tokens/s against 25.7 - because now there is a real draft distribution for the joint rule to work with. That combination has no measured cell below the ordinary rule at either temperature; its worst is -2%. So we re-ran the block-length sweep under it, overnight, four lengths at two temperatures with the noisy cells repeated three times.

Seven - the block the drafter was trained to predict, and the one we rejected a week ago - wins by 19% single-stream at temperature 0.7 and 16% at 1.0, and by 9% and 12% at two streams, for 2 to 5% given back at four and sixteen. Since the gateway runs one or two agents at a time, that is the trade we want, and it is the same trade we made last week when we chose five over four.

The convincing evidence is not the throughput, which is noisy on this box, but the acceptance: 3.85 to 4.83 tokens per step at seven against 3.18 to 3.92 at five, with six between the two and four below both. A clean ordering in the direction the mechanism predicts is worth more than any single fast run.

Finding 4: one run is not a measurement, and it nearly cost us

Probabilistic drafting alone produced 178.5 tokens/s at eight streams - its best cell, and 13% ahead of the combination we were about to ship. We flagged it in the merge request as a single run and therefore not to be trusted, and re-measured it overnight: 161.8, 167.8, 160.9, a mean of 163.5. The 178.5 was an outlier, the real gap is about 4%, and the decision it would have overturned was correct.

We are recording this because it is the second time in two articles that this box's variance has nearly written the wrong conclusion, and because the discipline that caught it is cheap: state the run count next to every number, and re-measure anything that would change a decision on its own.

What that means for our clients

• Answers are written two thirds faster than before any of this tuning. Single-stream at temperature 0.7, tokens arrive every 29 ms, against 35 ms after last week's change and 48 ms with the model's built-in guesser. The average response on our traffic is about 460 tokens: roughly 13 seconds, against 16 last week and 22 before.
• At the default temperature the gain is larger than last week's, not smaller. A client that sets nothing gets 28.6 tokens/s where last week's configuration gave 21.0. That is the case this change helps most, and it is the most common one.
• A busy box now shares the gain properly. With four to sixteen agents decoding at once each gets 19 to 31% more than the built-in head, where last week's block gave 7 to 13%.
• The first token still arrives at the same moment. Speculative decoding does not touch prompt processing, and it did not move: about 2,000 tokens/s throughout.
• Setting a temperature is still worth more than we can give you. A client that sends 0.7 for code instead of leaving the default 1.0 gets about 22% more decode, up from the 10% we quoted last week.
• The code is unchanged, by construction. Speculative decoding accepts exactly the tokens the big model would have produced anyway - though see the first caveat below, which is about how well we have checked that rather than whether it is true.

What it costs, and what we deployed

The draft's cached distribution is about 106 MiB, absorbed without changing anything else: the same 24.92 GiB weight load, the same seven-minute cold start. The deeper block carves more of the 36 GiB cache budget for its own use, so the budget holds 813,427 tokens instead of 838,501 - 3.10 concurrent 262K contexts instead of 3.20, a 3% trim. Prompt processing is unchanged.

Shipped: Qwen3.8-27B NVFP4 with a seven-token DFlash2 draft, draft_sample_method "probabilistic" and rejection_sample_method "block", on the same vLLM 0.28.0 image, same proxy, same served name, same 262K context. The two settings go together in the spec with a comment saying so, because either one alone is a regression somewhere.

The honest caveats

• We have not verified losslessness empirically. Speculative decoding accepts exactly the tokens the big model would have produced - that is a property of the construction, and it is why we keep saying quality is untouched. But we changed the sampling path and have confirmed only that chat, tool calls, image input and reasoning still work, not that the distribution of the output is unchanged. The benchmark run that would have checked it failed on a bug in our own evaluation harness, which had been collapsing per-task settings onto every task in a multi-task run; that is fixed now, and the check is still owed.
• The single-stream cells are wide. The three runs behind the served block's 34.8 were 39.7, 33.1 and 31.5. Every figure here is a mean with its run count, and the batched cells are two to three runs where last week's were often one.
• One box, one probe, one night. Same GB10, same checkpoint, same code-generation probe as both previous articles, each configuration a separate reboot.
• The drafter still does not match the weights it drafts for. DFlash2 was trained against Qwen's bf16 weights; we serve a community NVFP4 quantization, and the drafter reads internal states the quantization perturbs. Some of the single-stream spread is probably that. A draft trained on the quantized target remains the experiment we have not run, and it is now the only lever left that we know of.

What we learned that outlives the numbers

Two things, and neither is about this box.

The first is that a tuning result can carry a premise you did not write down. "Seven is too deep" was a fact about one drafting setting wearing the costume of a fact about hardware. It survived a week because nothing in how we recorded it made the premise visible, and the fix is not to measure more but to name the conditions a number depends on when you write it.

The second is that "lossless" and "free" are different words. Block verification changes nothing about what the model produces and still cost us 11% at eight streams, because a correctness guarantee says nothing about what the guarantee costs to compute. The estimate we got wrong last week priced a drafter by dividing its bytes by the bandwidth; the estimate we would have got wrong this week priced an acceptance rule by reading its proof.

CARouter Cloud is the part of CARouter we host ourselves: one ASUS Ascent GX10 (an NVIDIA GB10 "DGX Spark"-class box, 128 GB of unified memory, arm64) running vLLM behind a TLS proxy, serving two embedding models and, since the previous article, Qwen3.8-27B as the coding model. That article ended on a memory finding. This one is about speed, and it starts with a coincidence: on 2026-08-26 a ByteByteGo newsletter titled "How to Make LLMs 3X Faster", a clean explainer of speculative decoding, landed in the inbox, and the same morning vLLM 0.28.0 was published with the one drafter our model could not use yet. So we checked the article against the box line by line, then ran the experiment it had just made possible. Here is what got better, what did not, and what a "3×" headline is worth on unified memory.

The previous article: Laguna S 2.1 vs Qwen3.8-27B on CARouter's DGX Spark, speed lost to memory.

The short version

• Already there. Qwen3.8-27B was already running the article's technique: its built-in multi-token-prediction (MTP) head, re-run three times, was landing 2.76 tokens per step on production traffic - 75%, 57% and 44% at the three positions, read from 22 hours of the gateway's own requests.
• The lever. The one thing left was a better drafter. DFlash2, a five-layer block-diffusion draft for Qwen3.8-27B, needs vLLM 0.28.0; we bumped the image and measured MTP against DFlash2 at 1, 2, 4, 8 and 16 concurrent streams, at three draft lengths.
• The result. With a five-token block, each target step yields 3.45 accepted tokens against 2.55 (+35%), so decode is 28.5 tokens/s single-stream against 20.8 (+37%), 91 against 81 at four streams, 216 against 203 at sixteen. Where the gateway actually runs, one or two coding agents at a time, that is a third more per stream.
• The estimate we got wrong. We predicted 1.6-2.4× because the new draft reads 3.85 GB per pass. The step did not get faster at all: 121 ms before, 121 ms after. Every token gained came from more tokens accepted per step - 3.45 against 2.55, +35% - not from a faster step.
• The cost. 3 GiB of weights, a fifth of the KV budget (3.2 concurrent 262K contexts instead of 3.9), a seven-minute cold start, and vLLM's newer V2 model runner.
• Shipped. Qwen3.8-27B NVFP4 with a five-token DFlash2 draft on vLLM 0.28.0, all four containers on the new image. The MTP fallback is one line in one file.

What the article says, checked against the box

The article's argument is the standard one, and it is correct: generating a token means reading every weight once for a handful of arithmetic, so decode is bound by memory bandwidth and the math units idle; a small drafter can guess several tokens ahead, and the big model can check all of them in a single pass for about the price of one, keeping the guesses that match. The gain is the acceptance length - how many draft tokens survive per step - and it depends on the workload (code and extraction accept well, open-ended prose does not), on the sampling temperature (higher flattens the distribution and lowers acceptance), and on load (under a large batch the spare compute the trick spends is no longer spare). The article lists four places a draft can come from: a separate small model, prediction heads trained into the big one, a cheaper copy of the same model, and a search over the text already seen.

We went through that list against what was running. The prediction heads: that is Qwen's MTP head, already on, and vLLM's counters said what it was worth on real traffic - 208,647 tokens drafted, 122,270 accepted, 2.76 tokens per target step. The search over existing text (n-gram or suffix drafting): vLLM runs one proposer per engine, so it would replace the MTP head rather than add to it, and at 2.76 tokens a step the head is hard to beat on code the model has to write, as opposed to code it copies. A separate small model: a worse drafter than a head that reads the big model's own hidden states, and it costs memory. The cheaper-copy idea (QuantSpec): not in vLLM. The load cliff: the article says vLLM can disable speculation above a batch size, and it can - the modern form is a per-batch-size schedule of draft lengths - but on this box a step at our ceiling of 16 sequences carries at most 64 tokens, and a 27B on a GB10 is nowhere near compute-bound below a thousand, so we had no cliff to protect against.

What the check did surface is the shape of our traffic: 41 million prompt tokens for 192 thousand generated, 96% of the prompt tokens served from the prefix cache, which means a typical request spends about two seconds on the four thousand tokens it has to prefill and twenty-five on the 460 it generates. Decode is nine tenths of the wall clock. Speculation is exactly the right lever, and the only thing left to pull on it was the drafter.

The one thing left: a better drafter

DFlash2 (z-lab, ICML 2026) is a drafter trained for Qwen3.8-27B: five small transformer layers that read the big model's hidden states at five depths and, instead of guessing one token at a time, predict a whole block in one pass and let a small selector trace a path through the candidates. Its authors report 5.5 accepted tokens per step on GSM8K against 5.0 for the MTP head at the same draft length, and 3.4× over plain decoding. It ships as a 3.85 GB checkpoint and needs an architecture vLLM added in 0.28.0 - released, as it happened, that morning.

Before running it we wrote the estimate down, as we did last time. Our MTP step was ~123 ms, of which the three sequential re-runs of the single MTP layer were, by the earlier sweep's arithmetic, about 20 ms each. DFlash2 replaces them with one pass over 3.85 GB, which at this box's 273 GB/s is about 15 ms, and verifying eight positions instead of four is nearly free on a bandwidth-bound step. So: a ~100 ms step, and if acceptance went from 2.6 to somewhere between 3.5 and 5, single-stream decode would go from 18-20 tokens/s to 30-45. That was the prediction. Keep it in mind.

What we measured, and how

Same probe as the previous article, the bench script in our infrastructure repository: N concurrent code-generation requests through the production proxy, decode rate read from vLLM's own counters over a 40-second window, then a fresh 12K-token prefill. Two additions this time. The probe now takes the sampling temperature as a parameter, because the article's temperature point deserved a number: 0.7 is what every sweep so far used, 1.0 is the checkpoint's default and what the gateway sends when a client sets nothing. And single-stream points were repeated, because the first DFlash2 numbers spread more than we expected, and one 40-second window of one generation is not a measurement of a drafter whose acceptance depends on what the model happens to be writing.

Every configuration is one reboot of the same served checkpoint (unsloth's NVFP4 quantization of Qwen3.8-27B, 262K context, 36 GiB of KV) on the same box, the same day, one after the other.

Finding 1: the image bump alone changed nothing, and sixteen streams still scale

vLLM 0.28.0 ships the same CUDA 13.0.2, torch 2.13 and FlashInfer 0.6.16 stack as 0.27.1, picks the same CUTLASS NVFP4 kernel and the same FlashInfer attention on this silicon, and with the MTP head it decodes 20.8, 39.7, 81.2 and 124.4 tokens/s aggregate at 1, 2, 4 and 8 streams where 0.27.1 did 19.8, 40.0, 74.9 and 127.4 - noise. Prefill was 2,007 tokens/s against 2,083, one probe each. The one new cell is 16 streams: 202.7 tokens/s aggregate, 1.63× the eight-stream figure. The cliff the article warns about, where speculation starts costing more than it returns, is not there at our maximum batch.

Finding 2: 35% more tokens per step; the step did not get faster

DFlash2 at its native block of seven draft tokens: 26.0, 52.3, 81.0, 135.2 and 192.6 tokens/s aggregate at 1, 2, 4, 8 and 16 streams, against MTP's 20.8, 39.7, 81.2, 124.4 and 202.7. Up 25% and 32% at one and two streams, level at four, up 9% at eight, and down 5% at sixteen.

Then the number that explains it. At one stream DFlash2 accepted 3.16 tokens per step and decoded 26.0 per second: a 122 ms step. MTP accepted 2.55 and decoded 20.8: a 123 ms step. The step is the same. One pass of a five-layer drafter plus its selector plus an eight-position verify costs, on this box, exactly what three re-runs of a single MTP layer cost. Our estimate had priced the draft at its bandwidth, 3.85 GB in 15 ms, and forgot that the step also runs Qwen's Gated DeltaNet layers over eight query positions instead of four, the selector, and a verify that is only free while the batch is one. Everything DFlash2 gained, it gained by accepting more tokens per step; it gained nothing by being faster per step.

And the acceptance tells you where the block should end. On the probe DFlash2 lands 75%, 52% and 36% of its first three tokens, about what the MTP head lands on production traffic, then 28%, 22%, 15% and 12% for positions four to seven. Those four positions add 0.77 accepted tokens per step. At one or two streams they are free, because the step is bandwidth-bound and verifying four more positions costs no time; at sixteen streams a verify of eight positions across sixteen sequences is 128 tokens of real work per step, and it is what turned +9% at eight streams into −5% at sixteen.

Finding 3: seven is too deep, five is the block

So we measured shorter blocks. At five draft tokens: 28.5, 47.5, 91.4, 137.4 and 216.1 tokens/s across the five concurrencies, the one- and two-stream points averaged over two runs each. At four: 23.0, 47.8, 94.2, 142.4 and 229.7. The three DFlash2 blocks and the MTP head all run the same ~121 ms step; what differs is the yield - 2.55 tokens for MTP, 2.77 at four, 3.45 at five, 3.16 at seven, single-stream - and how much the verify costs once the batch grows. Five is the block that wins where the gateway lives, one and two coding agents at a time: +37% over MTP at one stream. And it keeps the gain at every batch size we can reach, +13% at four, +10% at eight, +7% at sixteen. Four edges it by 3-6% from four streams up and loses a fifth at one; seven wins nothing that five does not.

The article's "shrink the draft as concurrency rises" knob exists in vLLM as a per-batch-size schedule of draft lengths, and the scheduler honours it; but the DFlash2 speculator in 0.28.0 drafts a fixed block, so we did not try it blind on a production engine. A static five does the job on this box.

Finding 4: temperature is a real lever, and it lives on the client

The article says a higher sampling temperature lowers acceptance. It does, and it is not small. At the checkpoint's default temperature of 1.0 the MTP head accepts 43% instead of 52% and single-stream decode falls from 20.8 to 18.5 tokens/s: 11% for a parameter most clients never set. DFlash2 at that temperature is where the spread showed up. Four single-stream runs at the seven-token block gave 32.5, 26.8, 21.3 and 17.1 tokens/s, acceptance from 43% down to 16% on the same prompt family, a mean of 24.4 and a worst run below MTP's 18.5; the five-token block gave 19.8 and 22.1, the four-token block 22.3 and 20.3. MTP, whose head is steadier, spreads 18 to 21 across three days of measurements. A block drafter's acceptance depends on what the model happens to be writing in the window, which is why every DFlash2 number in this article is a mean with its spread and why the probe now repeats.

The practical reading is on the client side: a coding agent that sends temperature 0.7 for code instead of leaving the default gets a tenth of its decode back before any server change, and the gateway is one place that default could live.

What that means for our clients

Reduced to what a person using the coding endpoint sees:

• Answers are written about a third faster. Single-stream, tokens now arrive every 35 ms instead of 48. The average response on our traffic is about 460 tokens, so it completes in roughly 16 seconds instead of 22; a 2,000-token file rewrite takes about 70 seconds instead of 96.
• The first token arrives at the same moment as before. That wait is prompt processing, which speculative decoding does not touch: about two seconds per request on our traffic, because 96% of the prompt is already in the prefix cache from the previous turn.
• The gain shrinks when the box is busy, but it no longer disappears. With four to sixteen agents decoding at once, each gets 7 to 13% more than before instead of 37%. The native 7-token block lost ground at sixteen streams; the served one does not.
• The code is exactly the same. Speculative decoding only accepts tokens the model would have produced anyway, so quality, tool calls and image input are untouched, and the BigCodeBench pairing from the previous article still applies.
• Some responses feel faster than others. A drafter guesses well on predictable code (boilerplate, repeated patterns, edits that echo the file) and poorly on prose or unusual logic, so the speed-up varies with what is being written and averages out over a session.
• One thing is in the client's hands. An agent that sends a sampling temperature of 0.7 for code, instead of leaving the model's default of 1.0, gets roughly another 10%, because the draft is accepted more often when the model is less random.

What it costs

The draft is 2.95 GiB of weights (vLLM's loading line went from 21.97 to 24.92 GiB). Its own 2048-token sliding-window cache is carved from the same 36 GiB KV budget, which now holds 838,501 tokens instead of 1,014,892: 3.2 concurrent 262K-token requests instead of 3.9. Cold start is seven minutes instead of six. And 0.28.0 forces its V2 model runner for a DFlash2 draft, a newer code path, which served chat, separated the reasoning from the answer, parsed a tool call and read an image correctly in the same-day checks, and survived sixteen streams at seven draft tokens where the previous release had faulted at four streams with four MTP tokens.

What we deployed

Qwen3.8-27B NVFP4 with a five-token DFlash2 draft on vLLM 0.28.0; the two embedding servers on the same image; same proxy, same served name, same 262K context and 36 GiB of KV. The MTP head stays in the checkpoint and its configuration stays in the spec as one commented line, because a drafter whose acceptance swings between 16% and 43% on the same prompt deserves a fallback that swings less.

The honest caveats:

• DFlash2 was trained against Qwen's bf16 weights; we serve a community NVFP4 quantization, and the drafter reads hidden states the quantization perturbs. Some of the spread may be that. A draft trained on the quantized target is the experiment we have not run.
• Every figure is one box, one probe, one day. The one- and two-stream cells are means of two to four runs; the rest are single runs.
• The estimate was wrong by the whole cost of the draft pass. The lesson generalises: on unified memory, price a drafter by measuring its step, not by dividing its bytes by the bandwidth.
• Coding quality is untouched by construction - speculative decoding accepts exactly the tokens the big model would have produced - so the BigCodeBench pairing from the previous article stands.

What we got out of the day, beyond the number

A third more per stream is the headline, but the infrastructure came out of it with more than that: the image is on vLLM 0.28.0 with the upgrade procedure exercised end to end; the probe takes a temperature and repeats; the production counters now tell us the acceptance length of real traffic, which is the number to watch as the draft meets agents rather than a red-black-tree prompt; and forty-one evaluation sandboxes left over from the previous week's BigCodeBench run are gone from the box. The next lever is documented rather than guessed: block verification, a lossless acceptance rule the V2 runner exposes, and a drafter paired to the quantized weights.

CARouter routes AI requests to Canadian-served providers, and CARouter Cloud is the part we host ourselves: one ASUS Ascent GX10 (an NVIDIA GB10 "DGX Spark"-class box, 128 GB of unified memory, arm64) running vLLM behind a TLS proxy, joined to our Kubernetes cluster as capacity. It serves two embedding models and one coding model. This is the story of choosing the coding model, told with the numbers, because the number that decided it was not the one we expected.

The short version

• Speed. On our code-generation probe Laguna decodes 46.5 tokens/s single-stream and 176 aggregate at eight streams; Qwen 18.3 and 122. Laguna leads at every concurrency we tried.
• Reasoning traces. On the text a coding agent actually produces, the two are level per stream: about 17 tokens/s against 14.
• Memory. Qwen at 44 GiB resident leaves 62 GiB free; Laguna at 98 GiB leaves 8. That is 3.9 concurrent 262K-token requests against 1.2, and it is the number that decided it.
• Quality. 33 against 31 on BigCodeBench's first 50 problems, McNemar p = 0.73. Not a gap.
• Shipped. Qwen3.8-27B in a community NVFP4 quantization, MTP depth 3, 36 GiB of KV. Laguna stays one environment flip away.

The two candidates

Poolside Laguna S 2.1 was the incumbent: a 117B mixture-of-experts with 8.5B active parameters, served from Poolside's NVFP4 checkpoint with their DFlash speculative-decoding draft. It is a very good coding model, and it is a memory hog: ~98 GiB resident at 262K context, which on a 121.6 GiB box leaves room for the two embedders and nothing else.

Qwen3.8-27B was the challenger: a dense 27B, 64 layers of which 48 are Gated DeltaNet (linear attention) and 16 are full attention, native 262K context, a vision tower, and a multi-token-prediction (MTP) head trained into the checkpoint. Qwen's own card puts it near the frontier on agentic coding (SWE-bench Pro 61.7, Terminal Bench 2.1 73.0).

The naive expectation, which we wrote down before measuring: the dense 27B would be much slower. A GB10 has about 273 GB/s of memory bandwidth, decode is bandwidth-bound, and a dense 27B at FP8 reads ~27 GB of weights per token where Laguna's MoE reads ~4 GB. Roofline arithmetic said 10 tokens/s against 60. On a plain code prompt the arithmetic held up: Laguna is two and a half times faster single-stream and ahead at every batch size we tried. What the arithmetic missed is the workload a coding agent actually sends, and the memory the smaller model leaves behind - which is where the rest of this piece goes, and where the decision came from.

What we measured, and how

Every throughput figure here comes from one probe, the bench script in our infrastructure repository: N concurrent code-generation requests through the production proxy, decode rate read from vLLM's own token counters over a 40-second window, then one fresh 12K-token prefill. Same probe, same box, same day for both models, so the columns line up. Where a figure comes from the eval run instead - the decode rate on long reasoning traces, read from the same counters while BigCodeBench was running - it says so, because the two workloads do not give the same answer. Coding quality is BigCodeBench, the first 50 problems, uncapped output tokens, scored in a sandbox - the same 50 problems for both, so the two runs pair.

Finding 1: speculative decoding is worth 5 tokens a step on easy code and 1.5 on reasoning

Laguna's DFlash draft proposes 7 tokens per step. On our code-generation probe it lands 58% of them - 5.1 tokens per step - and Laguna decodes 46.5 tokens/s single-stream. On the long reasoning traces a coding agent actually produces (its BigCodeBench run, read from the same counters), acceptance falls to 21%, 1.48 tokens per step, and the per-stream rate to about 17. Qwen's single MTP layer, re-run three times, accepts about half at each position on either workload: 2.5 tokens per step, 18.3 tokens/s single-stream on the probe and about 14 per stream with eight eval requests in flight. Laguna's speculation is spectacular when the text is predictable and ordinary when it is not; Qwen's is steady, and it still started out slower, because its base step reads 27 GB.

So the lever for Qwen was the checkpoint. We swapped Qwen's official FP8 weights for a community NVFP4 quantization (unsloth's: MLPs at 4-bit, attention projections at FP8, KV scales and the MTP head included). vLLM put it on FlashInfer's CUTLASS NVFP4 kernel - the same kernel family that makes Laguna fast on this silicon - and the step time fell from 185 ms to 130 ms, more than the byte ratio alone predicts.

Twelve configurations later, the shipped one is NVFP4 at MTP depth 3. Depth 4 tied depth 3 single-stream and then crashed vLLM with an illegal memory access at four concurrent sequences, which is the kind of thing you only learn by running it. Larger prefill chunks made prefill a third slower. KV-cache precision turned out to be a calibration decision, not a speed one.

Finding 2: on throughput, Laguna wins the easy case and ties the hard one

A coding endpoint behind a gateway is never one request at a time, so we measured both models at 1, 2, 4 and 8 concurrent streams, same probe, same day.

Laguna leads at every count: 46.5, 52, 96 and 176 tokens/s aggregate against Qwen's 18.3, 40, 73 and 122. Qwen's curve is the straighter one - within 5% of linear to eight streams, because a dense model pays its weight read once per step whatever the batch, and 32 tokens per step is nowhere near compute-bound for a 27B. Laguna's flattens between one and two streams, where its draft's acceptance dips, then climbs 3.8x by eight. The crossover we expected, where batching would erode the MoE's active-parameter advantage, is not there by eight streams on this box.

What narrows the gap is the workload. On reasoning traces, read from the same counters during the BigCodeBench runs, Laguna does about 17 tokens/s per stream at two streams and Qwen about 14 per stream at eight (113 aggregate); Laguna above two streams on that workload is the one cell we did not measure. Per stream, on the text agents actually produce, the two are level; on code Laguna can predict, it is far ahead. Prefill is Laguna's at every length: 3,550 tokens/s against 2,088 on a 12K prompt, 2,836 at 55K and 1,634 at 200K.

Finding 3: 62 GiB free is four 262K sessions instead of one

This is the number that decided it. Our requirement was concrete: 262K context, and several coding clients in flight at once. Context lives in the KV cache, and the KV cache lives in whatever memory the weights leave behind.

Laguna at 98 GiB resident leaves 8 GiB free after the embedders. Its 12 GiB KV budget holds 1.2 full-context requests, and there is nowhere to grow it. Qwen at 44 GiB leaves 62 GiB free. Its per-token KV is actually pricier than Laguna's (32 KiB against 24, because its 16 attention layers use 256-wide heads), but it can afford a budget three times the size, with room for four.

At the 36 GiB of KV we deployed, Qwen holds nearly four concurrent 262K-token requests (3.9: 1,014,892 cached tokens, measured); at 48 it would hold five. Laguna holds one, with a second waiting. For a gateway whose coding use cases are agents reading whole repositories, that is not a throughput detail; it is the product.

Finding 4: quality is a tie, which is what we needed

On BigCodeBench's first 50 problems - uncapped output, scored in the same sandbox, the same 50 for both - Qwen solves 33 and Laguna 31. Paired problem by problem the two agree on 42 of them: 28 both solved, 14 both failed. Qwen alone solves five, Laguna alone three, and an exact McNemar test on those eight discordant problems gives p = 0.73. That is not a ranking, and it was never going to be one at n = 50; it is the absence of a gap, which is the result the memory finding needed to mean anything. What did differ is how long Qwen thinks. It produced 462K output tokens for the set against Laguna's 264K, and ten of its samples ran past 16K tokens where Laguna never crossed 11K - reasoning traces of 20 to 31K tokens on the hardest problems, none of them hitting the 32K cap. Those samples are also how we found two ten-minute timeouts, one in the proxy and one in the eval harness, that had been silently cutting long generations off and retrying them; both are 7200 s now, and every number in this article was measured after the fix.

Long-context retrieval held too. On the same needle-in-a-haystack grid that validated Laguna's 256K (four context lengths from 32K to 220K, three needle positions), Qwen scores 9.0/10 overall - perfect at 95K, 9.0 at 157K, and 8.0 at 220K where Laguna held a perfect 10. The deepest end of the window is where the incumbent keeps a small edge; everything below it is a draw.

What we deployed

Qwen3.8-27B, NVFP4, MTP depth 3, 262K context, 36 GiB of KV, images on, behind the same proxy under the same served name. Laguna stays on disk and in the repository, one environment flip away, because a decision made on n=50 and one weekend should stay reversible.

The honest caveats:

• The NVFP4 build is a community quantization, not Qwen's.
• 50 problems is a signal, not a leaderboard.
• Laguna's decode on reasoning traces was measured at two streams only, so its eight-stream figure on that workload is an inference, not a measurement.
• The throughput comparison reversed on us late: until Laguna went through the same probe we were setting its eval-run rate against Qwen's probe rate and calling Qwen 3.6x faster at eight streams, which it is not. The decision stands on the other three findings.
• Everything here is one box, so the batch curves are a property of dense-versus-MoE on unified memory, not a universal law.

What a second DGX Spark would change

The obvious question after a weekend like this is whether a second GB10 - two DGX Sparks, linked over their ConnectX-7 200 Gb/s ports, the configuration NVIDIA sells for 405B-class models - would have changed the decision. It would have changed the options more than the answer.

• Both models, no choice. The simplest use of a second box is to serve Laguna on one and Qwen on the other, each with its embedders, and let the gateway route by model. Two coding endpoints, two sovereign offers, and Laguna gets its whole 121 GiB back for KV it never had. That is the configuration we would run if a customer asked for Laguna specifically.
• Qwen twice, KV everywhere. Two independent Qwen boxes double the concurrent 262K capacity to ten, add failover, and keep the linear batch scaling on each. This is the boring answer and probably the right one for a gateway: capacity is additive, nothing crosses the interconnect on the request path.
• Tensor-parallel Qwen across both. vLLM can split the dense 27B over two nodes, which halves the weight bytes each box reads per step and, in principle, nearly doubles single-stream decode. The price is an all-reduce per layer over a 25 GB/s link; at Qwen's hidden size that is a few kilobytes per token per layer, so the cost is latency, not bandwidth - plausible, unmeasured, and it spends the second box on single-stream latency, the one column where Laguna is already far ahead on code and the two are level on reasoning.
• A bigger model instead. The genuinely new option: with 256 GB the candidates change. Laguna's main revision (99.7 GB, the eight BF16 expert layers Poolside kept for quality) fits with room for context; a 122B-A10B MoE at NVFP4 (83.5 GB on the Hub) would have Laguna's active-parameter economy with a stronger base and still leave well over 100 GB for KV across the pair. That is the experiment we would run next, and it is a different article.

What a second box would not change is the shape of the finding. Decode on this silicon is bound by bytes per step and helped by tokens per step; a gateway is served by aggregate throughput and concurrent context; and a dense model that fits in a third of the memory turns the remaining two thirds into exactly the thing a gateway sells. More boxes multiply that. They do not replace it.