Implicit Federated In-context Learning For Task-Specific LLM Fine-Tuning

TL;DR

The paper introduces IFed-ICL, a federated learning framework that enhances large language model fine-tuning using implicit representations, reducing computation and data transmission while improving task-specific performance.

Contribution

It proposes a novel implicit federated in-context learning method that avoids extensive parameter updates and enhances distributed task-specific model performance.

Findings

Achieves high performance on multiple text classification tasks.

Reduces data transmission and local computation compared to traditional fine-tuning.

Enables efficient distributed context learning with private data.

Abstract

As large language models continue to develop and expand, the extensive public data they rely on faces the risk of depletion. Consequently, leveraging private data within organizations to enhance the performance of large models has emerged as a key challenge. The federated learning paradigm, combined with model fine-tuning techniques, effectively reduces the number of trainable parameters. However,the necessity to process high-dimensional feature spaces results in substantial overall computational overhead. To address this issue, we propose the Implicit Federated In-Context Learning (IFed-ICL) framework. IFed-ICL draws inspiration from federated learning to establish a novel distributed collaborative paradigm, by converting client local context examples into implicit vector representations, it enables distributed collaborative computation during the inference phase and injects model residual streams to enhance model performance. Experiments demonstrate that our proposed method achieves outstanding performance across multiple text classification tasks. Compared to traditional methods, IFed-ICL avoids the extensive parameter updates required by conventional fine-tuning methods while reducing data transmission and local computation at the client level in federated learning. This enables efficient distributed context learning using local private-domain data, significantly improving model performance on specific tasks.