Blog

Nemotron 3.5 Bolt E4B vs Qwen3.6 35B-A3B: benchmark on a DGX Spark

NVIDIA's new Nemotron 3.5 Bolt E4B is a small model for voice assistants, and on a DGX Spark at 4 bits it weighs a third of the Qwen3.6 35B-A3B we serve. It calls tools as well and starts an assistant's answer sooner, but writes no faster for one request without its draft model and answers 56% of science questions against 83%.

ResearchBy Flavien Berwick
Research: Nemotron 3.5 Bolt E4B vs Qwen3.6 35B-A3B, benchmark on a DGX Spark. One request on llama.cpp, each with its draft model: 135 against 90 tokens a second. Tool calls right: 84% against 81%. Science questions right: 56% against 83%. Both at 4 bits on one NVIDIA DGX Spark.

NVIDIA released Nemotron 3.5 Bolt E4B on 23 October. It is Google's Gemma 4 E4B, trained further by NVIDIA for voice assistants, interactive characters and devices that have to answer quickly.

NVIDIA publishes it at 4 bits as a 5.8 GiB file for llama.cpp, the engine people use to run a model on their own computer. We ran it beside Qwen3.6 35B-A3B, the model behind CARouter Cloud's coding endpoint, on the chip of the NVIDIA DGX Spark, which is also the chip of our own server.

The last model we tested this way: LFM2.5 24B-A2B vs Qwen3.6 35B-A3B.

The short version

  • Size. At 4 bits Bolt weighs 5.8 GiB, against 19.0 GiB for Qwen in the same format.
  • Speed for one request. Small does not mean fast here. On llama.cpp Bolt writes 57 tokens a second and Qwen 65. With the draft model NVIDIA ships beside it, Bolt reaches 135, the fastest of everything we measured.
  • Speed for many. With sixteen requests at once, Bolt totals 445 tokens a second on llama.cpp against 229 for Qwen. Its draft model then costs more than it gives: 107.
  • An assistant's reply. With reasoning switched on, Bolt starts its answer after 1.5 seconds, or 0.8 with its draft, against 1.9 for Qwen as we serve it. Without reasoning, both start within a quarter of a second.
  • Quality. Bolt calls tools as well as Qwen, 84% against 81%, and follows instructions as well, 91% against 89%. It answers far fewer science questions, 56% against 83%, and codes a little worse.
  • What we decided. CARouter Cloud keeps Qwen3.6 35B-A3B for coding and agents. Bolt is a candidate for a different job: a fast, small model for assistants.

Small, but not light for each token

Bolt holds 8 billion parameters, and about 4 billion of them work on each token. NVIDIA calls that 4 billion effective. Qwen3.6 35B-A3B holds 35 billion, but only about 3 billion work on each token.

On the GB10, writing a token means reading the parameters that work on it from memory. Bolt reads more of them per token than Qwen does, so it is not faster for one request. It is much lighter to keep in memory: with sixteen conversations open on llama.cpp, Bolt held 7.9 GiB and Qwen 25.0 GiB.

Speed

One request alone on one GB10, tokens written per second. Nemotron 3.5 Bolt E4B on llama.cpp 57, with NVIDIA's draft model 135. Qwen3.6 35B-A3B on llama.cpp 65, with its own draft head 90. Qwen3.6 35B-A3B on vLLM as CARouter Cloud serves it 119.
One request alone on one GB10, tokens written per second. Nemotron 3.5 Bolt E4B on llama.cpp 57, with NVIDIA's draft model 135. Qwen3.6 35B-A3B on llama.cpp 65, with its own draft head 90. Qwen3.6 35B-A3B on vLLM as CARouter Cloud serves it 119.

A draft model is a small helper that guesses the next few tokens for the main model to check in one step. NVIDIA ships one for Bolt, and Qwen3.6 has its own built in. With its helper, Bolt is the fastest setup we measured for one request alone.

The gain depends on what is written. On our coding prompt the helper more than doubled Bolt's speed. On a short story it added much less: 67 tokens a second against 59, because prose is harder to guess.

Sixteen requests at once on one GB10, tokens written per second in total. Nemotron 3.5 Bolt E4B on llama.cpp 445, with NVIDIA's draft model 107. Qwen3.6 35B-A3B on llama.cpp 229, with its own draft head 305. Qwen3.6 35B-A3B on vLLM as CARouter Cloud serves it 473.
Sixteen requests at once on one GB10, tokens written per second in total. Nemotron 3.5 Bolt E4B on llama.cpp 445, with NVIDIA's draft model 107. Qwen3.6 35B-A3B on llama.cpp 229, with its own draft head 305. Qwen3.6 35B-A3B on vLLM as CARouter Cloud serves it 473.

With many requests the picture changes. Without its helper, Bolt totals 445 tokens a second with sixteen at once, nearly twice Qwen on the same engine.

With its helper it falls to 107. Bolt's helper pays only when a request runs alone, and NVIDIA's own example for it serves one request at a time. Qwen's draft head still helps with sixteen requests: 305 against 229.

Bolt also reads a prompt quickly: about 5 000 tokens a second on llama.cpp, against 2 900 for Qwen on the same engine and 6 500 on vLLM.

A voice assistant

Bolt is built for assistants, so we wrote one: a kitchen speaker for a family in Ottawa, with four tools for the weather, a timer, the lights and the shopping list. One person asked it twelve things, one after the other, and half of them needed a tool.

A voice assistant on one GB10: median seconds until the answer starts, over 12 turns. With reasoning off: Nemotron 3.5 Bolt E4B 0.13, with its draft 0.12, Qwen3.6 35B-A3B as CARouter Cloud serves it 0.21. With reasoning on: 1.52, 0.81 and 1.95.
A voice assistant on one GB10: median seconds until the answer starts, over 12 turns. With reasoning off: Nemotron 3.5 Bolt E4B 0.13, with its draft 0.12, Qwen3.6 35B-A3B as CARouter Cloud serves it 0.21. With reasoning on: 1.52, 0.81 and 1.95.

Both models picked the right tool, or no tool, in all twelve turns, with reasoning on and off. With reasoning off, both start answering within a quarter of a second, which no listener would notice.

With reasoning on, Bolt writes about 100 tokens per turn, its reasoning included, where Qwen writes about 215. Its answer starts in 1.5 seconds, or 0.8 with its helper. Qwen's starts in 1.9 seconds as we serve it.

Quality

Share of each test answered correctly, both models at 4 bits and reasoning, Nemotron 3.5 Bolt E4B then Qwen3.6 35B-A3B: coding 50% and 60%, following instructions 91% and 89%, calling tools 84% and 81%, science questions 56% and 83%, finding a fact in a long text 100% and 100%.
Share of each test answered correctly, both models at 4 bits and reasoning, Nemotron 3.5 Bolt E4B then Qwen3.6 35B-A3B: coding 50% and 60%, following instructions 91% and 89%, calling tools 84% and 81%, science questions 56% and 83%, finding a fact in a long text 100% and 100%.

On 500 requests from the Berkeley function-calling benchmark, Bolt got 84% right and Qwen 81%. The gap comes from requests that no tool fits: Bolt rightly called none 154 times out of 160, Qwen 135. With reasoning off Bolt still got 82%, while writing a quarter as many tokens.

IFEval gives 541 prompts with instructions a program can check, such as a word count or a required format. Bolt followed every instruction of 91% of them and Qwen of 89%, a gap chance can explain, and Bolt wrote six times fewer tokens. With reasoning off, Bolt dropped to 87%.

Knowledge is where the size shows. On 198 graduate-level science questions (GPQA Diamond), Bolt answered 56% correctly and Qwen 83%.

Coding needs a warning. With reasoning on, Bolt usually leaves out the line that opens a block of code, and keeps the line that closes it. A strict test runs that stray line as code and fails, which gives 6 of our 50 BigCodeBench problems.

With only that line removed, Bolt solved 25 of the 50 problems against 30 for Qwen. Any program that reads code out of Bolt's answers has to expect the missing line.

We also hid one sentence in documents of 16 000 to 52 000 tokens and asked for it back. Both models found it 12 times out of 12. Bolt cannot be asked about longer documents: its window stops at 65 536 tokens, against 262 144 for Qwen.

What we cannot tell you

  • How Bolt holds a long conversation or a character. NVIDIA's strongest results are there, and we tested single requests only.
  • How fast it is on vLLM, the engine CARouter Cloud serves with. NVIDIA publishes its 4-bit build for llama.cpp only.
  • How it does in French or in the other languages it was trained for. We tested English only.
  • Whether a gap of a few points on our smaller tests is real. Fifty coding problems cannot rank two models this close.

What this means for CARouter

CARouter Cloud keeps Qwen3.6 35B-A3B for coding and agents. Bolt reads a quarter of its context, answers far fewer questions of knowledge, and writes no faster for one request without its helper.

Bolt is still the first small model we measured that calls tools and follows instructions as well as Qwen, in a third of the memory, under a permissive licence. That makes it a candidate for a fast assistant model beside Qwen, rather than instead of it.