SHE-LoRA: Selective Homomorphic Encryption for Federated Tuning with Heterogeneous LoRA

TL;DR

SHE-LoRA introduces a privacy-preserving federated tuning method for large language models that balances security, efficiency, and performance by selectively encrypting model parameters based on sensitivity, suitable for heterogeneous device environments.

Contribution

The paper proposes a novel selective homomorphic encryption approach combined with LoRA for efficient, privacy-preserving federated LLM tuning in heterogeneous settings.

Findings

Maintains comparable performance to non-private models.

Achieves 99.71% reduction in communication overhead.

Reduces encryption time by 99.87% compared to baseline methods.

Abstract

Federated fine-tuning is critical for improving the performance of large language models (LLMs) in handling domain-specific tasks while keeping training data decentralized and private. However, prior work has shown that clients' private data can actually be recovered via gradient inversion attacks. Existing privacy preservation techniques against such attacks typically entail performance degradation and high costs, making them ill-suited for clients with heterogeneous data distributions and device capabilities. In this paper, we propose SHE-LoRA, which integrates selective homomorphic encryption (SHE) and low-rank adaptation (LoRA) to enable efficient and privacy-preserving federated tuning of LLMs in cross-device environments. Based on model parameter sensitivity assessment, heterogeneous clients adaptively negotiate and select a subset of model parameters for homomorphic encryption. To ensure accurate model aggregation, we design a column-aware secure aggregation method and customized reparameterization techniques to align the aggregation results with the heterogeneous device capabilities of clients. Extensive experiments demonstrate that SHE-LoRA maintains performance comparable to non-private baselines, achieves strong resistance to state-of-the-art attacks, and significantly reduces communication overhead by 99.71% and encryption time by 99.87%, compared to HE baselines.