Aggregating Low Rank Adapters in Federated Fine-tuning

TL;DR

This paper introduces a new aggregation method for low-rank adapters in federated fine-tuning of large language models, aiming to improve efficiency and performance on benchmark datasets.

Contribution

It proposes a novel aggregation technique for LoRA adapters in federated learning and compares it with existing methods on GLUE benchmarks.

Findings

The new aggregation method outperforms existing approaches in federated LoRA fine-tuning.

Federated LoRA fine-tuning achieves competitive results on GLUE benchmarks.

Parameter-efficient federated training reduces communication costs while maintaining performance.

Abstract

Fine-tuning large language models requires high computational and memory resources, and is therefore associated with significant costs. When training on federated datasets, an increased communication effort is also needed. For this reason, parameter-efficient methods (PEFT) are becoming increasingly important. In this context, very good results have already been achieved by fine-tuning with low-rank adaptation methods (LoRA). The application of LoRA methods in Federated Learning, and especially the aggregation of adaptation matrices, is a current research field. In this article, we propose a novel aggregation method and compare it with different existing aggregation methods of low rank adapters trained in a federated fine-tuning of large machine learning models and evaluate their performance with respect to selected GLUE benchmark datasets.