FedMCP: Parameter-Efficient Federated Learning with Model-Contrastive Personalization

TL;DR

FedMCP is a parameter-efficient federated learning method that uses lightweight adapters and model-contrastive regularization to personalize large language models across diverse clients, reducing communication costs and improving performance.

Contribution

The paper introduces FedMCP, a novel approach combining lightweight adapters and contrastive regularization for personalized federated learning of large language models.

Findings

Achieves significant performance gains over existing FL fine-tuning methods.

Reduces communication overhead by transmitting only lightweight adapters.

Effectively captures both universal and client-specific knowledge.

Abstract

With increasing concerns and regulations on data privacy, fine-tuning pretrained language models (PLMs) in federated learning (FL) has become a common paradigm for NLP tasks. Despite being extensively studied, the existing methods for this problem still face two primary challenges. First, the huge number of parameters in large-scale PLMs leads to excessive communication and computational overhead. Second, the heterogeneity of data and tasks across clients poses a significant obstacle to achieving the desired fine-tuning performance. To address the above problems, we propose FedMCP, a novel parameter-efficient fine-tuning method with model-contrastive personalization for FL. Specifically, FedMCP adds two lightweight adapter modules, i.e., the global adapter and the private adapter, to the frozen PLMs within clients. In a communication round, each client sends only the global adapter to the server for federated aggregation. Furthermore, FedMCP introduces a model-contrastive regularization term between the two adapters. This, on the one hand, encourages the global adapter to assimilate universal knowledge and, on the other hand, the private adapter to capture client-specific knowledge. By leveraging both adapters, FedMCP can effectively provide fine-tuned personalized models tailored to individual clients. Extensive experiments on highly heterogeneous cross-task, cross-silo datasets show that FedMCP achieves substantial performance improvements over state-of-the-art FL fine-tuning approaches for PLMs.