Revisiting Federated Fine-Tuning: A Single Communication Round is Enough for Foundation Models

TL;DR

This paper demonstrates that a single communication round in federated fine-tuning of foundation models achieves comparable performance to multiple rounds, significantly reducing communication costs and enabling more practical, privacy-preserving federated learning.

Contribution

The paper provides both theoretical and empirical evidence that one-shot federated fine-tuning is sufficient for large foundation models, challenging the necessity of multi-round aggregation.

Findings

One-shot federated fine-tuning achieves similar performance to multi-round methods.

Large foundation models benefit from lower training loss in one-shot settings.

Communication costs are significantly reduced with single-round aggregation.

Abstract

The recent advancement of foundation models (FMs) has increased the demand for fine-tuning these models on large-scale cross-domain datasets. To address this, federated fine-tuning has emerged, allowing FMs to be fine-tuned on distributed datasets across multiple devices while ensuring data privacy. However, the substantial parameter size and the multi-round communication in federated learning algorithms result in prohibitively high communication costs, challenging the practicality of federated fine-tuning. In this paper, we identify and analyze, both theoretically and empirically, that the traditional multi-round aggregation algorithms may not be necessary for federated fine-tuning large FMs. Our experiments reveal that a single round of aggregation (i.e., one-shot federated fine-tuning) yields a global model performance comparable to that achieved through multiple rounds of aggregation. Through rigorous mathematical and empirical analyses, we demonstrate that large FMs, due to their extensive parameter sizes and pre-training on general tasks, achieve significantly lower training loss in one-shot federated fine-tuning compared to smaller models. Our extensive experiments show that one-shot federated fine-tuning significantly reduces communication costs. It also has the potential to enable asynchronous aggregation, enhances privacy, and maintains performance consistency with multi-round federated fine-tuning on both text generation and text-to-image generation tasks. Our findings provide insights to revolutionize federated fine-tuning in practice, enhancing efficiency, reducing costs, and expanding accessibility for FMs.