CryptoGen: Secure Transformer Generation with Encrypted KV-Cache Reuse

TL;DR

CryptoGen introduces a scalable, privacy-preserving system for autoregressive Transformer generation that efficiently reuses encrypted KV-caches, significantly reducing latency while maintaining security in untrusted environments.

Contribution

It is the first system to enable scalable, privacy-preserving neural generation with encrypted KV-cache reuse, combining homomorphic encryption and secret sharing techniques.

Findings

Achieves 4.4x-7.6x lower per-token latency than existing systems.

Maintains near-linear latency and memory scaling for input lengths up to 512 tokens.

Demonstrates effectiveness on models trained on WikiText-2, PTB, and LAMBADA datasets.

Abstract

The widespread deployment of cloud-hosted generative models raises a fundamental challenge: enabling efficient autoregressive generation while preserving the privacy of both user prompts and model parameters in untrusted environments. We address this challenge in a client-server setting where an untrusted server hosts an autoregressive Transformer and the client requires cryptographic protection for both inputs and inference. We present CryptoGen, the first system to enable scalable privacy-preserving neural generation with persistent encrypted key-value (KV) cache reuse. Discriminative-task secure inference systems incur quadratic latency and memory growth when adapted to autoregressive decoding due to the lack of native encrypted KV-cache support. In contrast, CryptoGen achieves near-linear scaling by securely reusing and updating encrypted KV caches throughout generation. CryptoGen integrates homomorphic encryption and secret sharing to support both prefilling and generation. Key techniques include a unified encrypted KV-cache framework, heterogeneous SIMD encodings for different phases, optimized cipher-cipher matrix-matrix and matrix-vector operations, and efficient noise refresh and ciphertext concatenation mechanisms. Evaluation on generative Transformer models trained on WikiText-2, PTB, and LAMBADA shows that for input lengths of 128-512 tokens, CryptoGen achieves 4.4x-7.6x lower per-token latency than state-of-the-art discriminative secure inference systems, while maintaining near-linear latency and memory scaling, with advantages increasing for longer sequences. CryptoGen is released as an open-source library.