Answering the sovereignty concern and GPU shortage with homomorphic LLM inference
We tested client-assisted homomorphic inference from GPT-2 to Qwen3.6-35B-A3B, including a separate trusted CPU client. Methodology, measured throughput, privacy limits and lessons for private AI.
Could CARouter use GPUs operated by people we do not trust, while keeping customer prompts confidential? Homomorphic encryption makes that an appealing research question. We started with GPT-2 on an H100, then tested Qwen/Qwen3.6-35B-A3B on a Denvr A100 with 80 GB of GPU memory. The second experiment addresses a much more realistic model for an AI service.
Our interactive requirement was at least 10 output tokens per second per request. The validated 16-token encrypted-linear continuation reached 0.00598 tokens/s, with 35.5 minutes before the first token: about 1,673 times below the throughput target. A token is a small piece of text, often part of a word. We would not build an interactive private-chat service around this implementation.
Thank you to Denvr: its AI Ascend program granted CARouter $1,000 in compute credits, which made these experiments possible. The credits supported the modern-model GPU runs and the dedicated CPU controller. This is an acknowledgement of infrastructure support; the measurements and conclusions are ours.
The privacy problem starts in memory
Encryption protects a connection or a disk, but ordinary inference still handles readable prompts and conversation state in working memory. A conventional host administrator may inspect that memory, including the KV cache. Encrypting a volume during provisioning does not prevent this, and deleting a VM does not prove that its owner erased every copy.
Homomorphic encryption allows a worker to compute on encrypted values and return an encrypted result. Fully homomorphic encryption, or FHE, extends this idea to general computation. It requires a compatible computation engine: Terraform and dstack can manage VM lifecycles, but wrapping vLLM in an encrypted container does not make its arithmetic homomorphic.
From a small-model probe to a modern checkpoint
The initial H100 experiment used GPT-2 to establish that we could generate text through encrypted linear operations and check the results. Its best validated single-request variant, with prepared weights cached, reached 0.16 tokens/s and about 33.2 seconds to the first token. Those short warm runs were useful for testing the machinery, but GPT-2 does not represent the model quality customers expect today.
For the Denvr experiment we pinned the public BF16 checkpoint, tokenizer and software versions. We used PyTorch 2.9.1+cu128, Transformers 5.17.0 and Zama’s Concrete ML Extensions 0.1.9. The Zama library provided the encrypted linear operations; the remaining model computation ran on the client. We first checked that the real checkpoint loaded without missing or mismatched text-model parameters. Tiny random-model tests checked integration only; they contributed no target-model performance figures.
The model uses a mixture of experts: each token activates only selected expert networks. We intercepted functional linear products as well as ordinary linear modules, so routed expert projections and the output vocabulary projection were included. Attention, nonlinear operations, recurrent state, embeddings and sampling remained plaintext client work. This is client-assisted encrypted-linear inference, not a fully encrypted transformer.
How we measured it
We established a native vLLM serving baseline and a native same-framework generation baseline, then a clear quantized reference with the same arithmetic as the encrypted adapter. The adapter quantizes activations and weights to signed 8-bit values; the BF16 checkpoint label does not mean encrypted BF16 arithmetic. The serving baseline used greedy generation, thinking disabled and eager execution, with repeated short and long prompts and a concurrent group. Prefix reuse was possible; this was not maximum-capacity tuning.
For encrypted generation, public synthetic prompts contained 21 input tokens after chat formatting. The longer prompt asked why encryption matters; the shorter asked what two plus two equals. We measured time to the first token, subsequent decode time and whole-request duration separately. Decode tokens/s excludes the first token. Model loading, key generation, public-key upload and VM startup are outside those request timings.
The first placement put client and evaluator on the same GPU VM to measure computation. Its owner could read the client key and plaintext memory. We then ran a separate evaluator and a CPU client limited to 4 CPUs and 12 GiB of memory. The private key stayed in CPU-process memory; the GPU received a public compression key and encrypted activations. Attention and KV state stayed on the CPU VM.
Measured modern-model results
| Execution path | Output tokens | Output tokens/s | First token (s) |
|---|---|---|---|
| Native vLLM, short prompts | Up to 64 | 17.06 | 0.15 |
| Encrypted linear operations, same VM | 16 | 0.00598 | 2,132.13 |
| Encrypted linear operations, separate CPU client | 2 (including EOS) | 0.0056 | 2,231.13 |
The longer encrypted continuation completed and all its saved output scores matched the quantized reference exactly. It stopped at its output-length limit, partway through a paragraph; this was not a complete-answer quality evaluation. We deliberately stopped the additional repeat after independently validating the completed request. It contributes no extra throughput sample.
The separate-client trial returned “4” followed by the end-of-sequence token. Both count in its two-token output, leaving just one decode step after prefill. That step took 178.5 seconds. This is a network smoke measurement, not a long-conversation throughput estimate. All its saved scores also matched the reference exactly.
The native serving baseline reached about 17.06 tokens/s per request for short prompts. This establishes that the checkpoint is serviceable on the rented hardware; it is not an equal-arithmetic speed comparison with the quantized adapter. Neither native concurrent throughput nor synthetic matrix batching establishes encrypted model batch throughput, which we did not measure on this model.
The native rates are estimates from streamed content and reported token usage; its first-token column is time to first streamed content. The encrypted paths time each model forward pass and token selection. We retain the timing conventions and raw serving samples in our benchmark records.
Correctness came before speed
Our earlier precision investigation found settings that generated text but failed the numerical comparison. The modern-model runs inherited the accepted higher ciphertext-storage and key-switching precision. We checked encrypted integer products in the full-model audit, then saved and independently reloaded every completed encrypted request’s output-score tensors. The comparison tolerance remained 0.005; the observed maximum difference was zero.
This establishes agreement with the tested quantized reference. It does not establish equal answer quality to the original checkpoint: the native and quantized continuations differed. We did not run a task-quality suite or an independent cryptographic security review. Failed setup attempts, a serialization failure and deliberate interruptions remain in the experiment record rather than being counted as successful inference.
The network was not the main delay
The separate-client decode step spent 167.5 seconds inside the evaluator’s request timer. The difference between client round-trip time and that timer was 2.2 seconds, combining transport, serialization and scheduling. It is not pure network latency. Encrypted evaluation and weight handling dominated the measured delay; changing the VM orchestrator would not remove that work.
The network trial transferred 53.3 MB of application payload during that decode step, across 1,031 sequential linear calls. Initial public compression-key upload added 536.9 MB and took 2.09 seconds, separately from inference. These sizes exclude HTTP, SSH and TCP overhead. Comparing total time with the co-located smoke does not isolate networking either: client execution and integer-audit instrumentation also differed.
We separately tested caching prepared weights on the GPU. It improved some single-row component timings, but that microbenchmark was not a cached full-model generation run. An isolated kernel speedup cannot be presented as a new model throughput result.
What an answer costs
At the recorded GPU quote of $1.45/hour, the longer continuation consumed about $1.87 of measured request runtime. Its steady decode rate projects to roughly $67,394 per million output tokens, excluding prefill and setup. With the separate CPU allocation included, the short network request consumed about $1.24 of runtime. These are price-dependent calculations, not invoices or million-token capacity tests.
Estimated compute for the Denvr experiment, including failed attempts, provisioning and the controller’s lifetime, was $7.75. That estimate comes from allocation windows and hourly quotes; storage, networking, billing rounding and taxes are outside it. We did not consume the entire grant. The credits let us answer the engineering question before committing to a larger system.
What this means for private CARouter inference
For the tested engine and configuration, the interactive performance gate failed. That is evidence against building production orchestration around this implementation today, not proof that every homomorphic scheme or future optimization must be slow. A different engine must demonstrate useful speed, acceptable answer quality and its complete privacy boundary on the target workload.
The separate client also leaves explicit trust assumptions. Selected expert identifiers, tensor shapes and timing are visible to the evaluator. The ordinary CPU VM assumes its own infrastructure operator is trusted. Our synthetic prompts were public source fixtures, so the test does not demonstrate that their text was unknown to the GPU host. It does not make arbitrary community machines secure for customer prompts.
For native inference, a separate direction is confidential computing with compatible CPU and GPU protection, attestation and key release bound to the verified workload. We did not deploy or benchmark that architecture here. A GPU model name alone does not establish those protections. Confidentiality also does not attest a machine’s location or replace CARouter’s residency and provider-consent rules.
An independent CPU controller owned allocation deadlines and collected results, so a laptop disconnection did not control GPU billing. We retrieved the source and evidence, verified their hashes and confirmed through Denvr’s API that every experiment VM was deleted, including the credential-bearing controller. We retained output tensors for follow-up checks, not model weights or compiled binaries. API-confirmed deletion is an operational observation, not proof of physical erasure or protection against a host that previously saw plaintext.
Where the technology stands today
As of September 2026, research systems are tackling encrypted KV caches and hybrid execution combining homomorphic encryption with confidential computing. These are concrete paths forward, but results depend on model size, trust assumptions and whether a figure measures actual generation or projects it from components. We have not reproduced those systems, and their headline speeds cannot validate our workload.
For worldwide sovereignty concerns, encryption can reduce dependence on a remote operator’s ability or willingness to protect plaintext. It does not by itself establish sovereignty. Control of keys and the trusted client, permitted compute locations, metadata exposure, model integrity and the ability to keep serving remain separate requirements. A confidential calculation abroad does not automatically satisfy a customer’s residency policy.
For GPU scarcity, the relevant measure is useful answers per unit of compute. Encryption could make a larger pool of otherwise unsuitable machines usable, but its overhead must be smaller than the capacity that access unlocks. Our tested engine spends much more compute time per answer. Adding machines to this unchanged serial implementation would not demonstrate a faster individual conversation.
Our assessment is that the cryptographic building blocks are real, while a general-purpose, economically useful encrypted modern-LLM service on arbitrary community GPUs remains unproven by the evidence we evaluated. Today, our practical direction is trusted compute and separately evaluated attested confidential-computing deployments. We would revisit homomorphic community inference when a complete system demonstrates the required per-request speed, answer quality, protected state and metadata, and recoverable operation. That would address part of the sovereignty problem; it would still need to earn its place as a response to GPU shortages.
Request the benchmark files
These findings describe our September 2026 experiments. We retain the detailed measurements and source files for follow-up work. The earlier H100 experiment used a different model and hardware, so the results do not compare GPU generations.
Contact us to request the benchmark data and source files.