When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods

TL;DR

This paper explores the integration of parameter-efficient tuning methods with federated learning for NLP tasks, significantly reducing communication costs while maintaining performance, and provides a comprehensive empirical study and a new framework for this approach.

Contribution

It introduces a holistic empirical analysis of PETuning methods in FL and develops FedPETuning, a framework to facilitate research and application of PETuning in federated settings.

Findings

Communication overhead is significantly reduced with PETuning.

Lightweight model parameters maintain acceptable performance.

The framework supports various PETuning methods in FL scenarios.

Abstract

With increasing privacy concerns on data, recent studies have made significant progress using federated learning (FL) on privacy-sensitive natural language processing (NLP) tasks. Much literature suggests fully fine-tuning pre-trained language models (PLMs) in the FL paradigm can mitigate the data heterogeneity problem and close the performance gap with centralized training. However, large PLMs bring the curse of prohibitive communication overhead and local model adaptation costs for the FL system. To this end, we introduce various parameter-efficient tuning (PETuning) methods into federated learning. Specifically, we provide a holistic empirical study of representative PLMs tuning methods in FL. The experimental results cover the analysis of data heterogeneity levels, data scales, and different FL scenarios. Overall communication overhead can be significantly reduced by locally tuning and globally aggregating lightweight model parameters while maintaining acceptable performance in various FL settings. To facilitate the research of PETuning in FL, we also develop a federated tuning framework FedPETuning, which allows practitioners to exploit different PETuning methods under the FL training paradigm conveniently. The source code is available at \url{https://github.com/iezhuozhuo/FedETuning/tree/deltaTuning}.