FedBaF: Federated Learning Aggregation Biased by a Foundation Model

TL;DR

FedBaF is a novel federated learning method that securely integrates foundation models during aggregation, improving accuracy and reducing perplexity without exposing model weights, especially in non-IID and adversarial settings.

Contribution

Introduces FedBaF, a new federated learning approach that preserves foundation model confidentiality while enhancing training performance.

Findings

FedBaF outperforms traditional methods by up to 11.4% in IID settings.

FedBaF improves accuracy by up to 15.8% in non-IID scenarios.

Reduces language model perplexity by up to 39.2%.

Abstract

Foundation models are now a major focus of leading technology organizations due to their ability to generalize across diverse tasks. Existing approaches for adapting foundation models to new applications often rely on Federated Learning (FL) and disclose the foundation model weights to clients when using it to initialize the global model. While these methods ensure client data privacy, they compromise model and information security. In this paper, we introduce Federated Learning Aggregation Biased by a Foundation Model (FedBaF), a novel method for dynamically integrating pre-trained foundation model weights during the FL aggregation phase. Unlike conventional methods, FedBaF preserves the confidentiality of the foundation model while still leveraging its power to train more accurate models, especially in non-IID and adversarial scenarios. Our comprehensive experiments use Pre-ResNet and foundation models like Vision Transformer to demonstrate that FedBaF not only matches, but often surpasses the test accuracy of traditional weight initialization methods by up to 11.4% in IID and up to 15.8% in non-IID settings. Additionally, FedBaF applied to a Transformer-based language model significantly reduced perplexity by up to 39.2%.