# CARouter Blog

> Notes from the Canadian AI gateway: what we measured, what we changed, who we work with, and why.

Newest first. Each link is the article in Markdown; the same address without `.md` is its web page.

- In French: https://carouter.ai/fr/blog.md

- [LFM2.5 24B-A2B vs Qwen3.6 35B-A3B: benchmark on a DGX Spark](https://carouter.ai/blog/lfm2-5-24b-a2b-vs-qwen3-6-35b-a3b-on-a-gb10.md) (2026-10-19, Research, Flavien Berwick): Liquid AI says its new LFM2.5 24B-A2B is faster than the Qwen3.6 35B-A3B we serve. On a DGX Spark that holds on llama.cpp, 105 tokens a second against 65, but on vLLM, the engine we serve with, Qwen stays ahead at 118 against 96. At 4 bits LFM2.5 codes as well, follows instructions better and calls tools less reliably.
- [How we made our models serve +79% faster on Intel GPUs](https://carouter.ai/blog/qwen3-6-35b-a3b-79-percent-faster-on-intel-arc-pro-b60.md) (2026-10-12, Research, Flavien Berwick): Our two Intel Arc Pro B60 cards serve Qwen3.6 35B-A3B. Most of each step was being spent in Python on the server's processor, not on the cards. The engine now records the model's computation once and replays it: with two requests on a card, each one gets 64 tokens a second instead of 36, with four 55 instead of 35, and with eight 44 instead of 35, on the same hardware.
- [AirLLM runs Qwen3.6 35B-A3B at full precision on a 24 GB card: one token every 10 to 68 seconds](https://carouter.ai/blog/airllm-qwen3-6-35b-a3b-full-precision-on-a-24-gb-card.md) (2026-10-10, Research, Flavien Berwick): AirLLM is an open-source library that passes a model through a graphics card one layer at a time. With it, Qwen3.6 35B-A3B at full precision, a 67 GiB model, ran on a 24 GB card: in under 3 GiB for a short prompt, an image or a video, and with its whole window when the prompt is read in chunks. It writes one token every 10 seconds on a GB10 limited to 24 GB and one every 68 seconds on an Intel Arc Pro B60, where serving the model at 4 bits writes 60 to 75 tokens a second.
- [Qwen3.6 35B-A3B on two Intel Arc Pro B60 cards: 818 tokens a second in total](https://carouter.ai/blog/qwen3-6-35b-a3b-on-two-intel-arc-pro-b60.md) (2026-10-09, Research, Flavien Berwick): We tested a server with two Intel Arc Pro B60 cards. Running Qwen3.6 35B-A3B at 4 bits with one engine per card, it writes 818 tokens a second in total with 32 requests at once, and 558 with 16, where our GB10 writes 479. The GB10 stays faster for one request alone. This server will join our servers at CARouter Cloud.
- [Decision 2.0 and Vela 2.0 vs Kev-4B: benchmark on a GB10](https://carouter.ai/blog/decision-2-and-vela-2-vs-kev.md) (2026-10-07, Research, Flavien Berwick): vLLM Semantic Router has released two families of open decision models, Decision 2.0 and Vela 2.0. On a GB10, Vela 2.0 4B named the kind of task right on 96.6% of our 328 requests against 92.4% for Kev-4B, at three times the wait, and its 0.3B sibling reached 85.7% without a GPU.
- [Mistral Large 4 vs GLM-5.3: is Europe still in the game?](https://carouter.ai/blog/mistral-large-4-vs-glm-5-3-is-europe-still-in-the-game.md) (2026-10-06, Research, Flavien Berwick): Mistral's new flagship solved 56 of 148 hard coding problems against 62 for GLM-5.3, China's flagship. The gap is within chance, and Mistral Large 4 wrote seventeen times fewer tokens to get there.
- [Kolibri-1 vs Qwen3.6 35B-A3B: benchmark on a GB10](https://carouter.ai/blog/kolibri-1-vs-qwen3-6-35b-a3b-on-a-gb10.md) (2026-10-06, Research, Flavien Berwick): Aleph Alpha's new open-weight Kolibri-1 reads about 3 billion parameters per token, like the Qwen3.6 35B-A3B we serve. On a GB10 at 4 bits it solved 29 to 32 of our 50 coding problems against 33 for Qwen, in twice the memory and at half the speed.
- [Qwen3.6 35B-A3B is now 1.5x faster on CARouter Cloud](https://carouter.ai/blog/qwen3-6-mtp-head-one-and-a-half-times-faster.md) (2026-10-02, Product, Flavien Berwick): Our coding model shipped with a built-in helper that drafts the next tokens for the main model to check. We switched it on with one line of configuration: 115 tokens a second for one user instead of 75, and 479 instead of 342 with sixteen requests at once, on the same DGX Spark.
- [Pricing that makes sense](https://carouter.ai/blog/models-pricing-that-makes-sense.md) (2026-10-01, Product, Flavien Berwick): Our pricing policy for models we host ourselves: a market reference adjusted for sovereignty, explicit cost assumptions and room to invest while demand grows. Other providers keep their pricing.
- [Clef and Strands Decider vs Kev-4B: benchmark on a GB10](https://carouter.ai/blog/clef-strands-decider-vs-kev.md) (2026-10-01, Research, Flavien Berwick): Cloudflare's Clef and Strands Labs' Strands Decider are two new open decision models. On a GB10, Clef named the kind of task right on 94.5% of our 328 requests against 92.4% for Kev-4B, at five times the wait and nearly four times the memory, and OpenJev's untrained Qwen3.5-4B still leads at 97.0%.
- [OpenJev vs Kev-4B: benchmark on multiple CPU/GPU hardware](https://carouter.ai/blog/openjev-vs-kev.md) (2026-09-23, Research, Flavien Berwick): OpenJev asks an ordinary Qwen3.5-4B to pick an option letter, with no training for the job. On the two benchmarks we used to choose Kev-4B, it named the kind of task right on 95.7% of our 328 requests against 88.7% for Kev-4B, though it was confidently wrong far more often on public prompts.
- [Jev-class model Kev is available on CARouter (+ benchmarks)](https://carouter.ai/blog/kev-available-on-carouter.md) (2026-09-23, Product, Flavien Berwick): Kev 4B, an open decision model that answers typed questions the way TypeSafe's Jev does, is now callable at /v1/decisions from our own hardware in Canada. On requests held out from every design choice, it named the kind of task correctly 98.5% of the time, and picking a model with its answers came within two requests of the best single model at 25% less cost.
- [Smart routing with Kev: our benchmark](https://carouter.ai/blog/jev-kev-laya-decision-models.md) (2026-09-21, Research, Flavien Berwick): A request to CARouter could name no model at all and still land on the right one, at the price or speed you prefer. We tested three decision models that could make that call, on 1,020 requests in English and French. Kev is the strongest candidate; here is what deploying it would take.
- [Can CKKS unlock private AI?](https://carouter.ai/blog/ckks-gpu-benchmark.md) (2026-09-12, Research, CARouter): We tested current CKKS GPU libraries at Qwen model dimensions. Matrix speed and batching improved; a complete private chat service remains unproven.
- [Answering the sovereignty concern and GPU shortage with homomorphic LLM inference](https://carouter.ai/blog/homomorphic-encryption-community-gpus.md) (2026-09-12, Research, CARouter): We tested client-assisted homomorphic inference from GPT-2 to Qwen3.6-35B-A3B, including a separate trusted CPU client. Methodology, measured throughput, privacy limits and lessons for private AI.
- [Adding sticky caching to improve cost and speed](https://carouter.ai/blog/cache-affinity.md) (2026-09-10, Product, Flavien Berwick): Repeated instructions can cost less and process faster when an AI provider can reuse earlier work. Cache affinity helps preserve that opportunity after a provider change, while keeping your routing safeguards in place.
- [Greener tokens: the carbon footprint of every model, before you pick one](https://carouter.ai/blog/environment-indicator.md) (2026-09-07, Product, Flavien Berwick): Every offer on our documentation page now shows an estimate of the carbon emitted to serve a million tokens, coloured by the grid it runs on. Why we publish it, the two inputs and one formula behind it, and what it leaves out.
- [CARouter is in Denvr AI Ascend: cloud credits for benchmarks on Canadian hardware](https://carouter.ai/blog/denvr-ai-ascend.md) (2026-09-05, Partnerships, Flavien Berwick): CARouter is in Denvr AI Ascend, the startup program of a Canadian-owned cloud running its own GPUs in Calgary. The credits buy measurements on datacenter hardware, and those measurements feed the router. Denvr sells GPU hours and no inference endpoint, so nothing changes on the gateway.
- [Partnering with Augure: two Canadian-trained models, served only in Canada](https://carouter.ai/blog/augure-partnership.md) (2026-08-31, Partnerships, Flavien Berwick): Rosedale One and Tofino 3 are live on the gateway. Served in Canada or refused, never failed over abroad, prompts kept by nobody once served. Here is what stands behind the three chips on the card.
- [Qwen3.8-Flash-Next on CARouter's DGX Spark: better answers, for 10 times the wait](https://carouter.ai/blog/qwen3-8-flash-next-ten-times-the-wait.md) (2026-08-29, Research, Flavien Berwick): Qwen's Qwen4 architecture preview tops the coding leaderboards, and its smallest servable build is 125.9 GiB against a machine that has 121.6 GiB in total. 38% of it is a lookup table, so it runs anyway - streamed off the SSD, on a patched engine, alone on the box. It is about three times slower, holds a quarter of the conversations, and writes three and a half times as many tokens per problem. It probably codes better than what we serve, and on this hardware that advantage costs 13 minutes an answer.
- [Why we now serve Qwen3.6 35B-A3B instead of Qwen3.8 27B](https://carouter.ai/blog/qwen3-6-35b-a3b-instead-of-qwen3-8-27b.md) (2026-08-29, Product, Flavien Berwick): A dense 27B has to read all 23.4 GB of itself to write one token, which on our DGX Spark caps it at about 12 tokens a second and turned a routine agentic build into an hour of waiting. We replaced it with a model that is older, bigger and lower on the coding leaderboards - and reads 2.0 GB per token instead. It is 1.5x to 1.9x faster at every concurrency, prefills 3.2x faster, holds twice the 262K sessions, and on the fifty coding problems we run ourselves it scored 33/50 against 32/50.
- [Tuning speculative decoding on CARouter's DGX Spark: two thirds faster coding answers, same hardware](https://carouter.ai/blog/tuning-speculative-decoding-on-dgx-spark.md) (2026-08-27, Research, Flavien Berwick): Speculative decoding has a small draft model guess ahead so the big one can check several tokens at once. Which guesses it makes, and how they are checked, were two settings we had left at their defaults. Changing them took our self-hosted coding endpoint from 20.8 to 34.8 tokens a second per stream on the same box - and overturned the block length we had shipped a week earlier, which turned out to be a fact about the default setting rather than about the hardware.
- [MTP vs DFlash2 on CARouter's DGX Spark: 35% more tokens per step, not a faster step](https://carouter.ai/blog/dflash2-draft-on-dgx-spark.md) (2026-08-26, Research, Flavien Berwick): A newsletter promised 3× from speculative decoding the morning vLLM 0.28.0 shipped the drafter Qwen3.8-27B was missing. We checked the article against our 128 GB DGX Spark line by line, then measured DFlash2 against the MTP head at 1 to 16 streams and three block sizes. CARouter Cloud decodes a third faster per stream; the step got no faster at all; and the estimate we wrote down first was wrong by exactly the cost of the draft.
- [Laguna S 2.1 vs Qwen3.8-27B on CARouter's DGX Spark: speed lost to memory](https://carouter.ai/blog/one-gb10-two-coding-models.md) (2026-08-25, Research, Flavien Berwick): Poolside's Laguna S 2.1 decodes two and a half times faster than Qwen3.8-27B on our 128 GB DGX Spark. We shipped Qwen anyway: it fits in a third of the memory, and what it leaves free holds four coding sessions at 262K context where Laguna holds one. A weekend of measurements, with the charts, the mistakes and the config that won.