SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learning

TL;DR

SSFL introduces a method to identify and train sparse subnetworks at initialization in federated learning, significantly reducing communication costs and improving accuracy in non-IID data scenarios.

Contribution

The paper presents SSFL, a novel approach that finds sparse subnetworks before training, enhancing communication efficiency and accuracy in federated learning with non-IID data.

Findings

Achieves over 20% error reduction on CIFAR-10 compared to strong sparse baselines.

Reduces communication costs by 2x relative to dense federated learning.

Delivers over 2.3x faster communication time in real-world deployment.

Abstract

In this work, we propose Salient Sparse Federated Learning (SSFL), a streamlined approach for sparse federated learning with efficient communication. SSFL identifies a sparse subnetwork prior to training, leveraging parameter saliency scores computed separately on local client data in non-IID scenarios, and then aggregated, to determine a global mask. Only the sparse model weights are trained and communicated each round between the clients and the server. On standard benchmarks including CIFAR-10, CIFAR-100, and Tiny-ImageNet, SSFL consistently improves the accuracy sparsity trade off, achieving more than 20\% relative error reduction on CIFAR-10 compared to the strongest sparse baseline, while reducing communication costs by $2 \times$ relative to dense FL. Finally, in a real-world federated learning deployment, SSFL delivers over $2.3 \times$ faster communication time, underscoring its practical efficiency.