AirLLM runs Qwen3.6 35B-A3B at full precision on a 24 GB card: one token every 10 to 68 seconds

CARouter Cloud serves Qwen3.6 35B-A3B on its own hardware in Canada. We serve it at 4 bits, a compressed form of about 20 GiB that fits a 24 GB graphics card.

At full precision, the same model is 67 GiB. AirLLM is an open-source library that says it runs such a model on a small card without compressing it.

We tested that on this model. Does it run within 24 GB, with images, video and the model's whole window, and how fast?

The previous article: this model served at 4 bits on two Intel Arc Pro B60 cards.

The short version

• The result. AirLLM runs the full-precision model in under 3 GiB of a 24 GB card. Text, images and video work, and the answers are the ones our serving engine writes.
• The price. One token every 10 seconds on a GB10 limited to 24 GB, and one every 68 seconds on an Intel Arc Pro B60. A 500-token answer takes 1 hour 25 minutes, or 9 hours 27 minutes.
• Against serving. Served at 4 bits, the model writes 74.9 tokens a second on our GB10 and 59 to 62 on an Arc Pro B60. AirLLM is about 760 and 4 100 times slower.
• The window. As it ships, AirLLM reads a prompt of about 78 000 tokens within 24 GB, of the 262 144 the model accepts. With a short loop of our own around it, a prompt that nearly fills the window is read, in 15 minutes on the GB10 and in 47 on the Arc Pro B60.
• Our reading. It is a way to get one reference answer from the full model on a small card. It is not a way to serve.

How a 67 GiB model goes through a small card

A model is a stack of layers, 40 for this one, and each token is computed by going through them in order. A serving engine such as vLLM keeps every layer on the graphics card, which is why the card must be as large as the model.

AirLLM keeps the model on disk, one file per layer. For each token it copies a layer to the card, runs it, frees it, and takes the next one.

The card never holds more than one layer and the work in progress. In exchange, every token moves the whole model, 65 GiB, through the card.

What we ran

AirLLM 4.0.0 on Qwen's own full-precision files of Qwen3.6 35B-A3B, one request at a time, on two machines.

The first is a rented GB10, the chip of our own machine, with 128 GB of memory shared between the processor and the graphics. We limited the software to 24 GiB of graphics memory, so it behaves like a 24 GB card in a machine with plenty of memory.

The second is a real 24 GB card: one Intel Arc Pro B60 in the server of the previous article, which has 32 GB of memory. Its other card kept serving customers during the test.

AirLLM's one-line example did not load this model, which is not on its list. Its own class for a sibling model, Qwen3.8 27B, named by hand, did. We changed nothing in AirLLM.

It fits, and it answers like our engine

A short prompt, an image and a short video clip each peaked under 3 GiB of graphics memory. The Arc Pro B60's driver counted 3.5 to 4 GiB in use, of 24.

We sent the same prompts to vLLM holding the same full-precision files. The 24 tokens of text, the sentence about the image and the answer about the clip were identical, to the character.

One token every 10 seconds, or every 68

On the GB10 limited to 24 GB, AirLLM writes one token every 10 seconds. A 500-token answer, about a page, takes 1 hour 25 minutes. Our GB10 serves the 4-bit model at 74.9 tokens a second: the same answer in 7 seconds.

On the Arc Pro B60, one token takes 68 seconds and the answer 9 hours 27 minutes. The same card serves the 4-bit model at 59 to 62 tokens a second.

Full precision is not what costs the most. vLLM with the same full-precision files, on a GB10 using all its memory, writes 29.6 tokens a second: the answer in 17 seconds.

Almost all the time goes to moving the model. Of the GB10's 10 seconds, 5.7 bring the next layer onto the card, 4.1 free the previous one, and 0.3 is computing.

The two machines differ by where the files are. The GB10's memory holds all 65 GiB of them, so a layer is copied from memory. The server with the B60 has 32 GB, so every token reads the whole model from disk again, at about 1.1 GB a second.

The window: 78 000 tokens as shipped, all of it in chunks

The model accepts up to 262 144 tokens at once: its window. AirLLM gives the model the whole prompt in one pass, and the memory used grows with the prompt, by about 0.2 GiB per thousand tokens.

Within 24 GiB, a prompt of 77 923 tokens passed and one of 93 938 did not. An optional component that AirLLM's own code mentions, flash-linear-attention, moved the limit: 126 007 tokens passed and 198 087 did not.

On the Arc Pro B60, a prompt of 30 094 tokens passed in one pass. One of 61 908 tokens filled the card, and we stopped that run.

We then wrote a short loop around AirLLM. It gives the model the prompt 8 192 tokens at a time and keeps, between two passes, what the model retains of what it has read.

With it, a prompt of 260 199 tokens, nearly the whole window, was read in 15 minutes within 19.2 GiB on the GB10, and a fact placed at its start was found.

On the Arc Pro B60, the same prompt was read in 47 minutes, with 16.3 GiB of the card in use at the peak, and the fact was found too.

Each chunk is a full pass over the model. vLLM reads the same prompt in about a minute.

AirLLM's own 4-bit mode

AirLLM has a compression mode of its own, to 4 bits, announced as up to three times faster.

As shipped, it did not start on this model. Its check of free disk space asked for 238 GB for a 67 GiB model, and the option that skips the check stopped on its first file.

With that check bypassed by hand, it ran on the GB10 at one token every 129 seconds: 13 times slower than without compression.

We see only one use case for AirLLM: checking what a full-precision model answers to a given prompt, on a machine that cannot hold it.