Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsity

TL;DR

Comet is a private inference system for large language models that leverages activation sparsity prediction to significantly reduce communication and computation overhead in secure multi-party computation settings.

Contribution

It introduces a novel predictor for activation sparsity and a new private inference protocol that avoids zero computations, enhancing efficiency in privacy-preserving LLM inference.

Findings

Achieves 1.87x-2.63x speedup over state-of-the-art systems

Reduces communication by 1.94x-2.64x

Effectively exploits activation sparsity for privacy-preserving inference

Abstract

With the growing use of large language models (LLMs) hosted on cloud platforms to offer inference services, privacy concerns about the potential leakage of sensitive information are escalating. Secure multi-party computation (MPC) is a promising solution to protect the privacy in LLM inference. However, MPC requires frequent inter-server communication, causing high performance overhead. Inspired by the prevalent activation sparsity of LLMs, where most neuron are not activated after non-linear activation functions, we propose an efficient private inference system, Comet. This system employs an accurate and fast predictor to predict the sparsity distribution of activation function output. Additionally, we introduce a new private inference protocol. It efficiently and securely avoids computations involving zero values by exploiting the spatial locality of the predicted sparse distribution. While this computation-avoidance approach impacts the spatiotemporal continuity of KV cache entries, we address this challenge with a low-communication overhead cache refilling strategy that merges miss requests and incorporates a prefetching mechanism. Finally, we evaluate Comet on four common LLMs and compare it with six state-of-the-art private inference systems. Comet achieves a 1.87x-2.63x speedup and a 1.94x-2.64x communication reduction.