MIRA: A Method of Federated MultI-Task Learning for LaRge LAnguage Models

TL;DR

This paper presents MIRA, a federated multi-task learning method for large language models that uses parameter-efficient fine-tuning to improve local and global performance across clients.

Contribution

It introduces a novel federated multi-task learning approach for LLMs that incorporates client task structures and uses LoRA for efficient training.

Findings

Outperforms existing federated fine-tuning methods in local loss reduction.

Maintains comparable global performance across clients.

Reduces computational and communication costs with LoRA.

Abstract

In this paper, we introduce a method for fine-tuning Large Language Models (LLMs), inspired by Multi-Task learning in a federated manner. Our approach leverages the structure of each client's model and enables a learning scheme that considers other clients' tasks and data distribution. To mitigate the extensive computational and communication overhead often associated with LLMs, we utilize a parameter-efficient fine-tuning method, specifically Low-Rank Adaptation (LoRA), reducing the number of trainable parameters. Experimental results, with different datasets and models, demonstrate the proposed method's effectiveness compared to existing frameworks for federated fine-tuning of LLMs in terms of average and local performances. The proposed scheme outperforms existing baselines by achieving lower local loss for each client while maintaining comparable global performance.