Cachemir: Fully Homomorphic Encrypted Inference of Generative Large Language Model with KV Cache

TL;DR

Cachemir introduces a novel fully homomorphic encryption framework that efficiently integrates key-value cache mechanisms for secure, low-latency inference of large language models, enabling privacy-preserving generation.

Contribution

It presents new HE packing algorithms, an interleaved replicated packing method, and an augmented bootstrapping strategy tailored for KV cache integration in FHE-based LLM inference.

Findings

Achieves 48.83x and 67.16x speedup over prior methods on CPU.

Generates an output token in less than 100 seconds on GPU for Llama-3-8B.

Effectively reduces inference latency while maintaining privacy.

Abstract

Generative large language models (LLMs) have revolutionized multiple domains. Modern LLMs predominantly rely on an autoregressive decoding strategy, which generates output tokens sequentially and employs a key-value cache (KV cache) to avoid redundant computation. However, the widespread deployment of LLMs has raised serious privacy concerns, as users are feeding all types of data into the model, motivating the development of secure inference frameworks based on fully homomorphic encryption (FHE). A major limitation of existing FHE-based frameworks is their inability to effectively integrate the KV cache, resulting in prohibitively high latency for autoregressive decoding. In this paper, we propose Cachemir, a KV Cache Accelerated Homomorphic Encrypted LLM Inference Regime to overcome this limitation. Cachemir comprises three key technical contributions: 1) a set of novel HE packing algorithms specifically designed to leverage the computational advantages of the KV cache; 2) an interleaved replicated packing algorithm to efficiently compute the vector-matrix multiplications that result from using the KV cache in Transformer linear layers; and 3) an augmented bootstrapping placement strategy that accounts for the KV cache to minimize bootstrapping cost. We demonstrate that Cachemir achieves $48.83\times$ and $67.16\times$ speedup over MOAI (ICML'25) and THOR (CCS'25) respectively on CPU and consumes less than 100 seconds on GPU to generate an output token for Llama-3-8B.