Blog

LFM2.5 24B-A2B vs Qwen3.6 35B-A3B: benchmark on a DGX Spark

Liquid AI says its new LFM2.5 24B-A2B is faster than the Qwen3.6 35B-A3B we serve. On a DGX Spark that holds on llama.cpp, 105 tokens a second against 65, but on vLLM, the engine we serve with, Qwen stays ahead at 118 against 96. At 4 bits LFM2.5 codes as well, follows instructions better and calls tools less reliably.

ResearchBy Flavien Berwick
Research: LFM2.5 24B-A2B vs Qwen3.6 35B-A3B, benchmark on a DGX Spark. One request on llama.cpp: 105 tokens a second against 65. One request on vLLM: 96 against 118. Instructions followed: 95% against 89%. Both at 4 bits on one NVIDIA DGX Spark.

Liquid AI released LFM2.5 24B-A2B on 13 October. It holds 24 billion parameters and reads about 2.4 billion of them for each token it writes.

Liquid compares it with Qwen3.6 35B-A3B, the model behind CARouter Cloud's coding endpoint, and reports that it is faster and better at agent work. We measured both on GB10s, the chip CARouter Cloud runs on, at 4 bits as we serve our own models, to check both claims.

The last model we tested this way: Kolibri-1 vs Qwen3.6 35B-A3B.

The short version

  • Speed, on Liquid's engine. On llama.cpp, where Liquid measured its own figures, LFM2.5 writes 105 tokens a second for one request and Qwen 65. Liquid's claim holds.
  • Speed, on ours. On vLLM, the engine CARouter Cloud serves with, Qwen stays ahead: 118 against 96 for one request, 486 against 387 with sixteen at once.
  • Answers. LFM2.5 writes about 40% fewer tokens for the same answers. A typical coding answer arrives in 33 seconds against 40 for one user alone.
  • Quality. Level on coding, 30 of 50 problems each. Better at following instructions, 95% against 89%. Worse at calling tools, 70% against 81%, and at science questions, 73% against 83%.
  • Memory. LFM2.5 at 4 bits holds 12.9 GiB of weights, against 22.0 GiB for Qwen.
  • What we decided. CARouter Cloud keeps Qwen3.6 35B-A3B. LFM2.5 reads half as much context, takes no images, and calls several tools at once less reliably.

Two engines, two different races

llama.cpp is built to run a model on one computer for one person. vLLM is built to serve many people at once from one machine, and it is what CARouter Cloud runs.

Qwen3.6 also ships with a small helper that drafts the next few tokens for the main model to check, which makes it about half as fast again on vLLM. Liquid has announced a helper of its own for LFM2.5, for a different engine, and has not released it yet.

Which LFM2.5 we ran

Liquid publishes a 4-bit build of LFM2.5 for llama.cpp and an 8-bit one, in FP8, for vLLM. For vLLM at 4 bits, the format we serve Qwen in, we took two builds made by the community.

They taught us something about this model. Two fifths of the parameters LFM2.5 uses for each token sit outside its experts, the specialist networks that 4-bit builds usually compress. One build compresses only the experts and writes 58 tokens a second. The other compresses nearly everything and writes 96.

Same model, same engine, 1.65 times the speed. Compressing the rest cost nothing we could measure on code or instructions, and some accuracy on tool calls: 349 of 500 against 371. Below, LFM2.5 on vLLM is the faster build, the one we would serve.

Memory

At 4 bits LFM2.5's weights are 9 GiB lighter than Qwen's. With the same 28 GiB set aside for conversations, it holds 41.7 GiB in all against 52.6 GiB.

Per token of conversation the two cost about the same. LFM2.5 fits more conversations in that space only because each one can be half as long: 131 072 tokens against 262 144.

Speed

One request alone on one GB10, tokens written per second. On llama.cpp at Q4_K_M, the setup of Liquid AI's own figures: LFM2.5 24B-A2B 105, Qwen3.6 35B-A3B 65. On vLLM at 4 bits, how CARouter Cloud serves: LFM2.5 96, Qwen3.6 118.
One request alone on one GB10, tokens written per second. On llama.cpp at Q4_K_M, the setup of Liquid AI's own figures: LFM2.5 24B-A2B 105, Qwen3.6 35B-A3B 65. On vLLM at 4 bits, how CARouter Cloud serves: LFM2.5 96, Qwen3.6 118.

On llama.cpp, LFM2.5 is 1.6 times as fast as Qwen and reads a prompt 1.74 times as fast. Liquid reports 114 against 67 tokens a second on an AMD Ryzen AI Max+ 395, a machine with about the memory speed of the GB10. We found the same gap.

On vLLM the order flips. Qwen's helper puts it 20% to 28% ahead at every load. Without its helper, Qwen writes 76 tokens a second alone on this same machine, less than LFM2.5's 96.

Tokens written per second on one GB10, from one to sixteen simultaneous requests: Qwen3.6 35B-A3B on vLLM from 118 to 486, LFM2.5 24B-A2B on vLLM from 96 to 387, LFM2.5 on llama.cpp from 105 to 250, LFM2.5 in FP8 on vLLM from 54 to 173.
Tokens written per second on one GB10, from one to sixteen simultaneous requests: Qwen3.6 35B-A3B on vLLM from 118 to 486, LFM2.5 24B-A2B on vLLM from 96 to 387, LFM2.5 on llama.cpp from 105 to 250, LFM2.5 in FP8 on vLLM from 54 to 173.

llama.cpp does not share a machine well. With sixteen requests at once, LFM2.5 totals 250 tokens a second on it, against 387 on vLLM.

LFM2.5 reads prompts faster on vLLM too: about 9 900 tokens a second against 6 500 for Qwen. Liquid's FP8 build was the slowest of all on this chip, at 54 tokens a second alone.

Tokens a second are not the whole story, because LFM2.5 writes less. For the same 50 coding problems it wrote 173 thousand tokens and Qwen 291 thousand. A typical answer took 33 seconds for one user alone on vLLM, against 40 for Qwen.

Quality

Share of each test answered correctly, both models at 4 bits, LFM2.5 24B-A2B then Qwen3.6 35B-A3B: coding 60% and 60%, following instructions 95% and 89%, calling tools 70% and 81%, science questions 73% and 83%, finding a fact in a long text 75% and 100%.
Share of each test answered correctly, both models at 4 bits, LFM2.5 24B-A2B then Qwen3.6 35B-A3B: coding 60% and 60%, following instructions 95% and 89%, calling tools 70% and 81%, science questions 73% and 83%, finding a fact in a long text 75% and 100%.

We use the same 50 BigCodeBench problems for every model we consider: the model writes a Python function, and the function is run against tests. Both solved 30, and 26 of them were the same problems.

IFEval gives 541 prompts with instructions a program can check, such as a word count or a required format. LFM2.5 followed every instruction of 95% of them, Qwen of 89%. The gap is too large to be chance, and it is the one Liquid reports.

Calling tools goes the other way. On 500 requests from the Berkeley function-calling benchmark, LFM2.5 got 70% right and Qwen 81%. With one call per request LFM2.5 is as good. When a request needs several calls at once, it usually makes one, and it calls a tool more often when none fits.

On 198 graduate-level science questions (GPQA Diamond), LFM2.5 answered 73% correctly and Qwen 83%. Liquid's own table shows the same gap, and Liquid recommends pairing the model with web search for questions of knowledge. It is also the one test where LFM2.5 wrote more than Qwen.

We also hid one sentence, naming the first band to play on the Moon, in documents of 16 000 to 100 000 tokens, and asked for it back. Qwen found it 12 times out of 12, LFM2.5 9. Once LFM2.5 answered The Beatles. Its window also stops at 131 072 tokens, against 262 144 for Qwen, so it cannot be asked about longer documents at all.

What we cannot tell you

  • How LFM2.5 does inside an agent. Liquid trained it in OpenClaw, Hermes Agent, Pi and OpenCode, and we ran none of them.
  • How fast it gets with its helper. Liquid has not released it.
  • How it does in the fifteen other languages it was trained for. We tested English only.
  • Whether a gap of a few points on our smaller tests is real. Fifty coding problems cannot rank two models this close.

What this means for CARouter

CARouter Cloud's GB10 keeps Qwen3.6 35B-A3B. On the engine that serves our customers, LFM2.5 writes more slowly but writes less, codes as well and follows instructions better. It also reads half as much context, takes no images, calls several tools at once less reliably, and is licensed for commercial use only to companies under USD 10 million of annual revenue.

Liquid built it for one person on one machine, and there it does what Liquid says: 105 tokens a second on a GB10 with llama.cpp, 1.6 times Qwen.