Federated Fine-Tuning of Foundation Models via Probabilistic Masking

TL;DR

DeltaMask introduces a probabilistic masking approach for federated fine-tuning of foundation models, significantly reducing communication costs to below 0.1 bits per parameter while maintaining high performance across diverse datasets.

Contribution

The paper proposes DeltaMask, a novel stochastic masking method that enables ultra-low bitrate federated fine-tuning of foundation models, leveraging probabilistic filters and subnetworks.

Findings

Achieves as low as 0.09 bpp communication cost

Maintains high model performance across 8 datasets

Works with 5 different pre-trained architectures

Abstract

Foundation Models (FMs) have revolutionized machine learning with their adaptability and high performance across tasks; yet, their integration into Federated Learning (FL) is challenging due to substantial communication overhead from their extensive parameterization. Current communication-efficient FL strategies, such as gradient compression, reduce bitrates to around $1$ bit-per-parameter (bpp). However, these approaches fail to harness the characteristics of FMs, with their large number of parameters still posing a challenge to communication efficiency, even at these bitrate regimes. In this work, we present DeltaMask, a novel method that efficiently fine-tunes FMs in FL at an ultra-low bitrate, well below 1 bpp. DeltaMask employs stochastic masking to detect highly effective subnetworks within FMs and leverage stochasticity and sparsity in client masks to compress updates into a compact grayscale image using probabilistic filters, deviating from traditional weight training approaches. Our comprehensive evaluations across various datasets and architectures demonstrate DeltaMask efficiently achieves bitrates as low as 0.09 bpp, enhancing communication efficiency while maintaining FMs performance, as measured on 8 datasets and 5 pre-trained models of various network architectures.