Federated Adapter on Foundation Models: An Out-Of-Distribution Approach

TL;DR

This paper introduces FedOA, a federated learning method using adapter-based fine-tuning and regularization to improve out-of-distribution generalization for foundation models across diverse clients.

Contribution

It proposes a novel adapter-based approach with personalized regularization for OOD generalization in federated foundation models, supported by theoretical convergence analysis.

Findings

FedOA improves OOD generalization in federated NLP tasks.

Theoretical proof of convergence under non-convex settings.

Empirical validation on benchmark datasets shows enhanced performance.

Abstract

As foundation models gain prominence, Federated Foundation Models (FedFM) have emerged as a privacy-preserving approach to collaboratively fine-tune models in federated learning (FL) frameworks using distributed datasets across clients. A key challenge for FedFM, given the versatile nature of foundation models, is addressing out-of-distribution (OOD) generalization, where unseen tasks or clients may exhibit distribution shifts leading to suboptimal performance. Although numerous studies have explored OOD generalization in conventional FL, these methods are inadequate for FedFM due to the challenges posed by large parameter scales and increased data heterogeneity. To address these, we propose FedOA, which employs adapter-based parameter-efficient fine-tuning methods for efficacy and introduces personalized adapters with feature distance-based regularization to align distributions and guarantee OOD generalization for each client. Theoretically, we demonstrate that the conventional aggregated global model in FedFM inherently retains OOD generalization capabilities, and our proposed method enhances the personalized model's OOD generalization through regularization informed by the global model, with proven convergence under general non-convex settings. Empirically, the effectiveness of the proposed method is validated on benchmark datasets across various NLP tasks.