Nemotron 3.5 Lightning 30B-A3B vs Qwen3.6 35B-A3B: benchmark on a GB10
On a GB10 at 4 bits, NVIDIA's Nemotron 3.5 Lightning 30B-A3B matches the speed of the Qwen3.6 35B-A3B we serve, and its context cache holds 5.6 times as many tokens in the same memory. It codes and follows instructions as well, but calls tools less reliably, 76% against 81%, and answers 68% of science questions against 83%.
NVIDIA released Nemotron 3.5 Lightning 30B-A3B in August. It holds 30 billion parameters, and about 3 billion of them work on each token. Qwen3.6 35B-A3B, the model behind CARouter Cloud's coding endpoint, has the same shape.
NVIDIA publishes Lightning at 4 bits for vLLM, the engine we serve with, along with a small draft model recommended for the DGX Spark. We ran it on the GB10, the chip of the DGX Spark and of our own server, and compared it with our earlier results for Qwen.
The last model we tested this way: Nemotron 3.5 Bolt E4B vs Qwen3.6 35B-A3B.
The short version
- Speed for one request. The two are level. With NVIDIA's DSpark draft model, Lightning writes 116 tokens a second. Qwen writes 119 as we serve it.
- Speed for many. With sixteen requests at once, Lightning totals 489 tokens a second and Qwen 473, a gap too small to matter.
- Room for context. In the same memory, Lightning's context cache holds 7.0 million tokens against 1.25 million for Qwen.
- Quality. Lightning codes and follows instructions as well as Qwen. It calls tools less reliably, 76% against 81%, and answers fewer science questions, 68% against 83%.
Speed
A draft model is a small helper that guesses the next few tokens for the main model to check in one step. Lightning can use three of them, and NVIDIA recommends DSpark on a DGX Spark. Our results agree: DSpark is 1.66 times as fast as no draft at all, ahead of the other two.
DSpark keeps its whole gain under load, 1.66 times again with sixteen requests. In our last test, Bolt E4B's draft model lost three quarters of the speed at the same load.
Both models read a prompt at about 6 500 tokens a second. Qwen's figures come from our earlier benchmark. We ran Qwen again briefly on this test machine, and it came within 4% of those figures, so the two sets can be compared.
Room for long conversations
A server keeps the context of each conversation in a cache, and that cache limits how many long requests can run at once. We gave both models 28 GiB. With requests of 262 144 tokens, the window we serve Qwen with, that holds 27 requests for Lightning and 5 for Qwen.
Two things explain the gap. Only 6 of Lightning's 52 layers are attention layers, which have to remember every token, and its 23 Mamba layers keep a small state of fixed size instead. NVIDIA's build also stores the cache in 8 bits, where Qwen's stores it in 16.
This is the count the engine gives when it starts. We did not fill the cache with real conversations, so it remains a ceiling until someone does.
A short agent turn
We repeated our kitchen speaker test: four tools, and twelve requests from one person, half of which need a tool. With reasoning on, Lightning with its draft starts its answer after 1.1 seconds and Qwen after 1.9. Lightning reasons in about 120 tokens, where Qwen uses about 215.
With reasoning on, both models picked the right tool, or no tool, in all twelve turns. With reasoning off, both start within a quarter of a second, but three of Lightning's four setups made one mistake: they said the office lights were off without calling the tool.
Quality
On 50 BigCodeBench problems, Lightning solved 32 and Qwen 30. On IFEval's 541 prompts, Lightning followed every instruction of 492 and Qwen of 484. Chance can explain both gaps.
Tool calling is a real difference. On 500 requests from the Berkeley function-calling benchmark, Lightning got 381 right and Qwen 403, and chance is an unlikely explanation.
Most of the gap sits in one category, the requests that need several different tools at once: Lightning got 19 of 44 and Qwen 35. In 22 of them, Lightning made one call and stopped. An agent could collect the other calls on its next turn, but this test reads a single turn.
Science needs a warning. The test asks for the answer as "ANSWER: $LETTER", and Lightning copies the dollar sign and adds a second one: "ANSWER: $D$". The scorer reads that as no answer, which gives 12 of 198.
With the dollar signs accepted, Lightning answered 134 of 198 correctly and Qwen 165, whose score does not change. A program that asks Lightning for this format has to expect the habit.
Lightning also reasons for longer on these questions. It reached our limit of 32 768 tokens on 30 of them without answering, against 7 for Qwen, and we count those as wrong. NVIDIA reports 76% on this test for the same build.
We also hid one sentence in documents of 16 000 to 100 000 tokens and asked for it back. Qwen found it 12 times out of 12 and Lightning 10. Both of Lightning's misses ran out of the 8 192 tokens we allow while still reasoning.
What we cannot tell you
- How the two compare on long agent work. NVIDIA's own table gives Qwen 70% on SWE-bench Verified against 53% for Lightning, and we tested single requests only.
- How many long conversations Lightning really holds under load. The cache figure is the engine's count, not a measurement of real traffic.
- How it behaves beyond 262 144 tokens. NVIDIA validates it up to one million, and we did not test that.
- How it does on llama.cpp. We measured the 4-bit build on vLLM only.
- How it does in French or in the other languages it was trained for. We tested English only.
- Whether a gap of a few points on our smaller tests is real. Fifty coding problems cannot rank two models this close.