Blog

Kolibri-1 vs Qwen3.6 35B-A3B: benchmark on a GB10

Aleph Alpha's new open-weight Kolibri-1 reads about 3 billion parameters per token, like the Qwen3.6 35B-A3B we serve. On a GB10 at 4 bits it solved 29 to 32 of our 50 coding problems against 33 for Qwen, in twice the memory and at half the speed.

ResearchBy Flavien Berwick
Research: Kolibri-1 vs Qwen3.6 35B-A3B, benchmark on a GB10. Weights in memory: 42.7 GiB against 21.9. One request alone: 53 tokens a second against 117. Coding problems solved: 29 to 32 against 33 of 50. Longest conversations held in the same 28 GiB: 9.9 against 4.8.

Aleph Alpha released Kolibri-1 on 3 October: an open-weight model from Germany, under the Apache 2.0 licence, built for German and English. It holds 78 billion parameters and reads about 3.5 billion of them for each token it writes.

That last figure is why we looked at it. Qwen3.6 35B-A3B, the model behind CARouter Cloud's coding endpoint, reads about 3 billion parameters per token out of 35 billion. The two do a similar amount of work per token, and one of them is more than twice the size. We ran both on the same GB10 to see what the extra size costs and what it buys.

An earlier article: why we serve Qwen3.6 35B-A3B.

The short version

  • Memory. At 4 bits, Kolibri-1's weights take 42.7 GiB against 21.9 GiB for Qwen. As Aleph Alpha ships it, in FP8, they take 73.5 GiB and the model holds 103 GiB of a 121.6 GiB machine.
  • Speed. One request alone gets 53 tokens a second from Kolibri-1 and 117 from Qwen. With sixteen requests at once the totals are 290 and 486.
  • Coding. On our 50 coding problems, Kolibri-1 solved 29 to 32 depending on the build, and Qwen solved 33. Fifty problems cannot separate those scores.
  • Long conversations. Kolibri-1 remembers them at half the cost: the same 28 GiB holds 9.9 conversations of 262,144 tokens against 4.8 for Qwen.
  • What we decided. We are not adding Kolibri-1 to CARouter Cloud's machine. It would take twice the memory to answer coding requests about as well, at half the speed.

Two models that read the same amount per token

Both models are mixtures of experts. Each layer holds hundreds of small specialist networks, and each token is sent to a handful of them. All of them must sit in memory, and only the chosen few are read.

Kolibri-1 keeps 384 experts per layer and uses 6 of them per token. Qwen3.6 35B-A3B keeps 256 and uses 8. The work per token is close, so the speed should be close, and the memory should not be.

Which Kolibri-1 we ran

Aleph Alpha publishes Kolibri-1 in FP8, which is 8 bits per weight. We serve our own models at 4 bits, and Aleph Alpha publishes no 4-bit build, so we took two that the community made in the first days:

  • The most downloaded one, a Q4_K_M file in the GGUF format. It runs on llama.cpp, with a patch its author ships because llama.cpp does not know this architecture yet.
  • An NVFP4 build. It runs on vLLM, the software we serve with, in the same 4-bit format as our Qwen. This is the like-for-like comparison.

We also ran the FP8 release itself, to tell the model apart from its compression. Each model ran alone on the machine, with its maker's default settings, reasoning on.

Memory

Memory held on one GB10 with the same 28 GiB set aside for conversations: 52.1 GiB for Qwen3.6 35B-A3B at 4 bits, 73.5 GiB for Kolibri-1 at 4 bits and 103.1 GiB for Kolibri-1 in FP8, out of 121.6 GiB.
Memory held on one GB10 with the same 28 GiB set aside for conversations: 52.1 GiB for Qwen3.6 35B-A3B at 4 bits, 73.5 GiB for Kolibri-1 at 4 bits and 103.1 GiB for Kolibri-1 in FP8, out of 121.6 GiB.

Kolibri-1 at 4 bits needs 21 GiB more than Qwen, all of it weights. On a machine that also serves embeddings, a reranker and our decision model, that is the room those other models live in.

The FP8 release fits, and nothing else does. It left 10 GiB free on the whole machine.

Kolibri-1 wins the other half of the memory question. A model also has to remember each conversation in progress, and most of Kolibri-1's layers only look at the last 512 tokens. From the same 28 GiB it holds 2.59 million tokens of conversation against 1.25 million for Qwen.

Speed

Tokens written per second on one GB10, from one to sixteen simultaneous requests: Qwen3.6 35B-A3B from 117 to 486, Kolibri-1 NVFP4 from 53 to 290, Kolibri-1 Q4_K_M on llama.cpp from 64 to 228, Kolibri-1 FP8 from 49 to 161.
Tokens written per second on one GB10, from one to sixteen simultaneous requests: Qwen3.6 35B-A3B from 117 to 486, Kolibri-1 NVFP4 from 53 to 290, Kolibri-1 Q4_K_M on llama.cpp from 64 to 228, Kolibri-1 FP8 from 49 to 161.

Qwen is between 1.6 and 2.2 times as fast at every load. Part of that is a feature Kolibri-1 does not have: Qwen ships with a small helper that drafts the next tokens for the main model to check, which makes it half as fast again on this machine. Without the helper Qwen wrote 75 tokens a second alone, still ahead of Kolibri-1's 53.

A typical answer to one of our coding problems took Qwen 37 seconds for one user alone. The same problem took Kolibri-1 about 91 seconds.

Reading the prompt is a tie. Both models take in a 12,000-token prompt at about 6,400 to 6,800 tokens a second on vLLM.

The GGUF build is a different story on a different engine. Alone it is the fastest Kolibri-1, at 64 tokens a second. It shares the machine badly, with irregular totals between two and eight requests, and it reads a prompt three times slower. Going from 8 bits to 4 bits also made Kolibri-1 faster on vLLM, by up to 80% under load.

Quality

Coding problems solved out of 50: Qwen3.6 35B-A3B 33, Kolibri-1 NVFP4 29, Kolibri-1 Q4_K_M 32, Kolibri-1 FP8 31. Tokens written to answer them: 262 thousand, 337 thousand, 438 thousand and 412 thousand.
Coding problems solved out of 50: Qwen3.6 35B-A3B 33, Kolibri-1 NVFP4 29, Kolibri-1 Q4_K_M 32, Kolibri-1 FP8 31. Tokens written to answer them: 262 thousand, 337 thousand, 438 thousand and 412 thousand.

We use the same 50 BigCodeBench problems for every model we consider: the model writes a Python function, and the function is run against tests. Qwen solved 33, as it did on our own machine in August.

Kolibri-1 solved 32 in the Q4_K_M build, 29 in NVFP4 and 31 in FP8. The models agree on most problems: Q4_K_M and Qwen gave the same result on 47 of the 50. None of these gaps is statistically meaningful on 50 problems.

Kolibri-1 thinks longer to get there. It wrote between 1.3 and 1.7 times as many tokens as Qwen for the same 50 answers, and tokens are what a client pays for and waits for.

We also hid one sentence in documents of up to 220,000 tokens and asked for it back. Every model found it every time it finished its answer. Four times out of forty-eight, a model spent its whole answer allowance on reasoning first: Qwen once, Kolibri-1 three times.

Aleph Alpha's own published results point the same way. They put Kolibri-1 slightly ahead of Qwen3.6 35B-A3B on one coding benchmark and behind it on another.

What we cannot tell you

  • How Kolibri-1 does in German. It is the model's first language and its main argument, and we did not test it. We did not test French either.
  • Whether a gap of a few problems is real. Detecting a difference that small would take several hundred problems.
  • How an official 4-bit build would behave. The two we ran were made by the community, without Aleph Alpha's involvement.
  • What Kolibri-1 does beyond 262,144 tokens. Aleph Alpha validates it up to a million.
  • How it scores with more reasoning. We left the reasoning effort at its default, and Aleph Alpha's published results use the highest setting.

What this means for CARouter

CARouter Cloud's machine keeps Qwen3.6 35B-A3B. For coding in English, Kolibri-1 gave us about the same answers for twice the memory and more than twice the wait.

That is a verdict on one machine and one use. Kolibri-1 is an Apache 2.0 model trained from scratch in Europe, it handles long conversations cheaply, and its German is the part we did not measure. A workload in German, or a provider with more memory per machine, would weigh these numbers differently.