Can CKKS unlock private AI?
We tested current CKKS GPU libraries at Qwen model dimensions. Matrix speed and batching improved; a complete private chat service remains unproven.
Our first experiment generated text through encrypted linear operations, but missed our interactive target by a wide margin. This follow-up asks whether current CKKS GPU libraries provide better building blocks for private AI.
Read the first homomorphic inference benchmark.
On an A100 80 GB SXM, a packed 512 → 2,048 projection took 23.43 ms with FIDESlib and 9.89 ms with Cheddar. Batching also increased aggregate throughput. These are validated matrix measurements: we did not generate a complete LLM answer with CKKS, and neither setup demonstrated confidentiality from its host.
Why test CKKS?
CKKS is the Cheon–Kim–Kim–Song scheme for approximate arithmetic on encrypted numbers. It can pack values into one ciphertext and operate on them together. That makes it relevant to the matrix operations inside neural networks, while introducing numerical error that must be measured. The original CKKS paper.
We built pinned revisions of FIDESlib and Cheddar. FIDESlib offers GPU CKKS operations with OpenFHE interoperability. Cheddar provides a GPU implementation with optimized linear transforms. This changes the arithmetic engine from our first experiment; it is not an encryption setting that can be enabled in vLLM. FIDESlib. Cheddar.
What we actually measured
The screen used dimensions from Qwen3.6-35B-A3B, with public synthetic vectors and public synthetic weights. Input values were sampled uniformly from −1 to 1; weights used the same range divided by the square root of the input width. We measured encrypted-vector × plaintext-matrix products, with prepared weights reused across trials.
Every configuration had 1 warm-up followed by 3 measured trials. We synchronized the GPU around each evaluation and compared every output against a plaintext reference, with a fixed maximum absolute error of 0.0001. The figures below show medians; the matrix chart and table also show the observed minimum and maximum, not confidence intervals.
GPU evaluation excludes key generation, weight preparation, encoding, encryption, decryption and network transport. Both cryptographic clients ran on the GPU VM. The separate CPU controller managed collection and deletion; it was not a separate trusted cryptographic client. These placements were suitable for public benchmark inputs, not private customer prompts.
The FIDES matrix configuration enforced OpenFHE’s HEStd_128_classic parameter check, with ring dimension 16,384, depth 2 and 40-bit scaling moduli. Cheddar used its upstream bootparam_40 parameters at level 2, with ring dimension 65,536 and 32-bit arithmetic. The parameter sets and evaluation paths differ, so this compares our configurations rather than isolating a library-only speedup.
Packing matters as much as the library
We first padded rectangular matrices to square layouts. A denser layout reduced unused work by packing diagonals and combining partial sums. In FIDES, the 512 → 2,048 projection fell from 83.13 ms to 23.43 ms. The table uses this denser layout for both libraries. Cheddar uses the alternate minimum-key-switch path that passed our numerical checks.
| Input → output width | FIDESlib | Cheddar |
|---|---|---|
| 512 → 2,048 | 23.43 (23.4–23.46) | 9.89 (9.88–9.89) |
| 2,048 → 512 | 24.37 (24.32–24.38) | 10.07 (10.07–10.09) |
| 2,048 → 1,024 | 48.64 (43.82–49.6) | 14.33 (14.32–14.36) |
| 2,048 → 2,048 | 83.15 (82.87–90.52) | 23.28 (23.28–23.28) |
| 4,096 → 2,048 | 83.94 (83.69–86.56) | 23.5 (23.47–23.51) |
| 2,048 → 4,096 | 82.38 (82.05–84.04) | 23.29 (23.29–23.31) |
We also fetched one real gate-projection matrix from the pinned checkpoint, without downloading the full model. FIDES evaluated that 2,048 → 512 projection in 23.74 ms, with maximum absolute error below 0.000001 in the measured trials. The activations remained synthetic. This checks one real-weight operation, not a transformer block, generated text or model quality.
Batch throughput is not chat speed
For Cheddar’s 2,048 → 1,024 projection, packing 16 independent vectors into one ciphertext kept GPU evaluation near 14.4 ms. GPU-only throughput rose from 69 to 1,111 vectors/s. All vectors used the same public matrix and encryption context; this was not a test of batching unrelated customers with different keys.
Including encoding, encryption, decryption and decoding, the measured batch-16 component took 63.23 ms and delivered 253 vectors/s. Preparing its reusable weights took 52.3 seconds and is excluded from those timings. We have not demonstrated that every prepared weight in the full model can remain cached, or measured network and queueing costs.
The rejected trials matter
Our first Cheddar fused matrix-transform configuration returned incorrect results, despite accurate encryption/decryption and a correct single rotation. A non-BSGS configuration and the alternate minimum-key-switch path passed the small check. We used the latter for the reported screen. We did not establish the root cause of the rejected path or claim that every upstream configuration fails.
We also tested FIDES bootstrapping, which refreshes a ciphertext to allow further computation. The 40-bit configuration failed decryption. Higher-precision configurations returned maximum errors of about 0.000207 and 0.000127, both above our fixed 0.0001 gate. These trials stopped during warm-up; we do not publish their durations as validated bootstrap performance.
Cheddar’s supplied client explicitly warns that it is for testing and is not secure. It keeps test secrets on the GPU. FIDES’s parameter check is useful, but our co-located client still exposed its private key to the VM owner. Passing a numerical test does not validate a deployment’s privacy. Cheddar’s client warning.
What this means for the 10 tokens/s target
Our target allows 100 ms per generated token. For scale only, serially summing the measured Cheddar medians for the shared-expert gate, up and down projections across 40 model layers gives 1.2 seconds of GPU work. This illustrative sum excludes routed experts, attention, nonlinear operations, conversation state, the vocabulary projection and client costs. It is not measured model throughput or a universal lower bound: fusion, batching and a different implementation could change it.
We therefore stopped at the component screen instead of claiming a full CKKS inference benchmark. There is no CKKS tokens/s or cost-per-million-tokens result from this experiment. For a batched projection alone, GPU evaluation cost per million vectors is 1,000,000 × P × t / (3,600 × B), where P is the GPU hourly price, t is evaluation time in seconds and B is batch size. Setup, client work, transport and the rest of the model are additional.
Sovereignty and spare GPUs: a research path, not a service yet
Computing without giving a GPU operator readable prompts could eventually widen the pool of usable hardware across the world. This experiment shows that current CKKS libraries merit further work on that path. It does not establish a practical solution to GPU shortages or a complete sovereignty guarantee. More cryptographic computation can also consume more GPU capacity per useful answer.
A usable system still needs a trusted client and key boundary, validated encrypted attention and state, workable nonlinear approximations, acceptable answer quality, and a policy for leakage through expert selection and traffic patterns. Confidentiality alone also cannot force a host to return a correct answer, keep serving or erase its copies. Terraform and dstack can manage machines; these properties require a designed and tested inference protocol.
Further homomorphic work would require a complete validated encrypted block with a separate trusted client, followed by model-quality and end-to-end latency tests if it passes. We are deferring that work. The present evidence does not justify launching encrypted community-GPU chat at our interactive target.
A possible follow-up: confidential VMs with protected GPUs
Could an encrypted virtual machine make community GPU inference practical? The distinction is what the encryption protects. Encrypted disks protect stored files, while an ordinary VM or an added type-2 hypervisor still leaves live memory accessible to the outer host. A hardware-backed confidential VM can protect CPU memory, but ordinary GPU passthrough does not extend that protection to GPU memory and computation.
The architecture worth evaluating combines a confidential CPU VM, a GPU running in a supported confidential-computing mode, and a protected connection between them. An independent verifier would check fresh CPU and GPU attestation evidence, the approved guest workload and its session identity before releasing keys or sending private prompts. GPU passthrough is part of that setup; it is not the security guarantee by itself. Confidential VM and GPU reference architecture.
Unlike homomorphic evaluation, this would let a conventional engine such as vLLM compute on plaintext inside the protected environment. It avoids expanding model operations into homomorphic arithmetic, but its performance on our workload remains to be measured. Trust moves to the hardware security boundary, firmware, approved guest software and attestation and key-release services.
Compatible community hosts could therefore contribute without receiving readable prompts through their normal host administration tools. This does not make arbitrary hardware suitable: the platform must support and expose the complete protected path. The owner can still interrupt service or retain ciphertext copies, and the guarantee does not cover every physical attack or side channel. Confidentiality also does not establish geographic residency or sovereignty by itself. Confidential-computing threat model and limits.
This is a proposed follow-up, deferred for now. We have not run this confidential-VM experiment and report no throughput or security-validation result for it. If we revisit it, we would first verify attestation and refusal of unsupported configurations, then compare the same model and workload with and without confidential mode, measuring first-token latency, decode speed and cost.
Denvr’s compute credits supported this work. The measurements and conclusions are ours. The worker and controller were deleted after the results and source archives were retrieved and verified; the downloaded Qwen matrix and compiled libraries were not retained.