Plug-and-Play Parameter-Efficient Tuning of Embeddings for Federated Recommendation

TL;DR

This paper introduces a parameter-efficient fine-tuning framework for federated recommendation systems that reduces communication costs by minimizing embedding parameters while maintaining or improving accuracy.

Contribution

It proposes a novel PEFT-based embedding method for federated recommendation, integrating techniques like LoRA, Hash encoding, and RQ-VAE, with seamless compatibility and improved efficiency.

Findings

Significantly reduces communication overhead in federated recommendation.

Improves recommendation accuracy with fewer embedding parameters.

Demonstrates effectiveness across various models and datasets.

Abstract

With the rise of cloud-edge collaboration, recommendation services are increasingly trained in distributed environments. Federated Recommendation (FR) enables such multi-end collaborative training while preserving privacy by sharing model parameters instead of raw data. However, the large number of parameters, primarily due to the massive item embeddings, significantly hampers communication efficiency. While existing studies mainly focus on improving the efficiency of FR models, they largely overlook the issue of embedding parameter overhead. To address this gap, we propose a FR training framework with Parameter-Efficient Fine-Tuning (PEFT) based embedding designed to reduce the volume of embedding parameters that need to be transmitted. Our approach offers a lightweight, plugin-style solution that can be seamlessly integrated into existing FR methods. In addition to incorporating common PEFT techniques such as LoRA and Hash-based encoding, we explore the use of Residual Quantized Variational Autoencoders (RQ-VAE) as a novel PEFT strategy within our framework. Extensive experiments across various FR model backbones and datasets demonstrate that our framework significantly reduces communication overhead while improving accuracy. The source code is available at https://github.com/young1010/FedPEFT.