Low-Parameter Federated Learning with Large Language Models

TL;DR

This paper introduces LP-FL, a low-parameter federated learning approach for large language models that reduces costs and improves performance in few-shot natural language understanding tasks.

Contribution

LP-FL combines prompt learning, iterative soft-labeling, and LoRA techniques to enable efficient federated learning with large language models in resource-constrained environments.

Findings

LP-FL outperforms full-parameter federated learning in sentiment analysis.

LP-FL matches or exceeds centralized training performance in few-shot scenarios.

LP-FL demonstrates robustness against overfitting in federated settings.

Abstract

We study few-shot Natural Language Understanding (NLU) tasks with Large Language Models (LLMs) in federated learning (FL) scenarios. It is a challenging task due to limited labeled data and communication capacities in FL, especially with mobile devices. Recent studies show LLMs can be prompted to perform few-shot NLU tasks like sentiment analysis and arithmetic reasoning. However, the huge sizes of LLMs result in high computation and communication costs, making classical FL schemes impractical. To address these challenges, we propose Low-Parameter Federated Learning (LP-FL). LP-FL combines few-shot prompt learning from LLMs with efficient communication and federating techniques. Our approach enables federated clients to assign soft labels to unlabeled data using gradually learned knowledge from the global model. Through iterative soft-label assigning, we continually expand the labeled set during the FL process. Additionally, to reduce computation and communication costs, LP-FL utilizes the Low-Rank Adaptation (LoRA) technique for compact learnable parameter construction, efficient local model fine-tuning, and affordable global model federation. LP-FL consistently outperforms Full-Parameter Federated Learning (FP-FL) in sentiment analysis tasks across various FL settings. Its resistance to overfitting allows LP-FL to equal or surpass centralized training in few-shot scenarios.