FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients

TL;DR

FedRA introduces a simple, flexible federated tuning method that allocates model layers randomly to heterogeneous clients, enabling effective fine-tuning even when clients have limited resources, demonstrated by superior results on large-scale image datasets.

Contribution

FedRA presents a novel random allocation strategy for federated tuning that seamlessly integrates with transformer models and supports resource-constrained clients without model modifications.

Findings

FedRA outperforms existing methods on large-scale image datasets.

Supports scenarios where clients cannot support the entire global model.

Easily integrated into existing transformer-based models.

Abstract

With the increasing availability of Foundation Models, federated tuning has garnered attention in the field of federated learning, utilizing data and computation resources from multiple clients to collaboratively fine-tune foundation models. However, in real-world federated scenarios, there often exist a multitude of heterogeneous clients with varying computation and communication resources, rendering them incapable of supporting the entire model fine-tuning process. In response to this challenge, we propose a novel federated tuning algorithm, FedRA. The implementation of FedRA is straightforward and can be seamlessly integrated into any transformer-based model without the need for further modification to the original model. Specifically, in each communication round, FedRA randomly generates an allocation matrix. For resource-constrained clients, it reorganizes a small number of layers from the original model based on the allocation matrix and fine-tunes using adapters. Subsequently, the server aggregates the updated adapter parameters from the clients according to the current allocation matrix into the corresponding layers of the original model. It is worth noting that FedRA also supports scenarios where none of the clients can support the entire global model, which is an impressive advantage. We conduct experiments on two large-scale image datasets, DomainNet and NICO++, under various non-iid settings. The results demonstrate that FedRA outperforms the compared methods significantly. The source code is available at \url{https://github.com/leondada/FedRA}.