Bridging the Gap Between Foundation Models and Heterogeneous Federated Learning

TL;DR

This paper introduces RaFFM, an adaptive framework that compresses foundation models for resource-efficient federated learning, enabling high-performance AI tasks across heterogeneous edge devices.

Contribution

The paper proposes novel model compression algorithms tailored for federated learning, facilitating the deployment of large foundation models on resource-constrained edge devices.

Findings

RaFFM significantly improves resource utilization efficiency.

Target models achieve comparable performance to full-sized models.

Effective across NLP and computer vision tasks.

Abstract

Federated learning (FL) offers privacy-preserving decentralized machine learning, optimizing models at edge clients without sharing private data. Simultaneously, foundation models (FMs) have gained traction in the artificial intelligence (AI) community due to their exceptional performance across various tasks. However, integrating FMs into FL presents challenges, primarily due to their substantial size and intensive resource requirements. This is especially true when considering the resource heterogeneity in edge FL systems. We present an adaptive framework for Resource-aware Federated Foundation Models (RaFFM) to address these challenges. RaFFM introduces specialized model compression algorithms tailored for FL scenarios, such as salient parameter prioritization and high-performance subnetwork extraction. These algorithms enable dynamic scaling of given transformer-based FMs to fit heterogeneous resource constraints at the network edge during both FL's optimization and deployment stages. Experimental results demonstrate that RaFFM shows significant superiority in resource utilization efficiency and uses fewer resources to deploy FMs to FL. Despite the lower resource consumption, target models optimized by RaFFM achieve performance on par with traditional FL methods applied to full-sized FMs. This is evident across tasks in both natural language processing and computer vision domains.