Collaborative and Efficient Personalization with Mixtures of Adaptors

TL;DR

This paper introduces FLoRAL, a federated learning framework that efficiently personalizes models for clients using low-rank adaptors, improving generalization and robustness in data-scarce scenarios.

Contribution

FLoRAL is a novel parameter-efficient method that personalizes federated models through client-specific mixtures of low-rank adaptors, casting it as a multi-task learning problem.

Findings

FLoRAL outperforms full model mixtures in data-scarce settings.

FLoRAL provides better personalization than locally tuned adaptors.

Theoretical analysis shows improved gradient variance reduction.

Abstract

Heterogenous data is prevalent in real-world federated learning. We propose a parameter-efficient framework, Federated Low-Rank Adaptive Learning (FLoRAL), that allows clients to personalize in groups by mixing between low-rank adaptors, where the mixtures are client-specific. FLoRAL is a model parameterization that casts personalized federated learning as a multi-task learning problem, with weight sharing as an implicit regularizer. It is memory-efficient, as the personalized parameters (i.e., base model + adaptors) are all federated. Our results show that FLoRAL can generalize better than a mixture of full models when data are scarce. It can also consistently personalize better than models with a locally tuned adaptor per client. This demonstrates the benefits of "federated personalization" and its robustness against overfitting. We derive the convergence rates and show theoretically that FLoRAL can lead to better variance reduction of the base model's gradients.