HE-LRM: Efficient Private Embedding Lookups for Neural Inference Using Fully Homomorphic Encryption

TL;DR

HE-LRM introduces optimized techniques for privacy-preserving neural inference with encrypted embedding lookups, significantly improving efficiency and enabling practical deployment of encrypted DLRMs and other models.

Contribution

This work presents novel embedding compression and packing strategies that enable efficient FHE-based inference for models with large sparse embeddings, a previously underexplored area.

Findings

Achieved 56× speedup over state-of-the-art embedding compression.

Demonstrated end-to-end encrypted DLRM inference with practical latency.

Applied embedding-lookup primitives to large language models.

Abstract

Fully Homomorphic Encryption (FHE) allows for computation directly on encrypted data and enables privacy-preserving neural inference in the cloud. Prior work has focused on models with dense inputs (e.g., CNNs), with less attention given to those with sparse inputs such as Deep Learning Recommendation Models (DLRMs). These models require encrypted lookup into large embedding tables that are challenging to implement using FHE's restrictive operators and introduce significant overhead. In this paper, we develop performance optimizations to efficiently support embedding lookups in FHE-based inference pipelines. First, we present an embedding compression technique using client-side digit decomposition that achieves a 56$\times$ speedup over state-of-the-art. Next, we propose a multi-embedding packing strategy that enables ciphertext SIMD-parallel lookups across multiple tables. Crucially, our goal is not only to retrieve the correct embeddings, but to do so in a way that produces ciphertext outputs in a layout that is directly compatible with downstream encrypted computations server-side. We name our approach HE-LRM and demonstrate end-to-end encrypted DLRM inference. We evaluate HE-LRM on UCI (health prediction) and Criteo (click prediction), achieving inference latencies of 24 and 489 seconds, respectively, on a single-threaded CPU. Finally, while our evaluation focuses on DLRMs, we investigate and apply our embedding-lookup primitives to other models such as LLMs, which require both batched and single-embedding lookups.