Sparse-ProxSkip: Accelerated Sparse-to-Sparse Training in Federated Learning

TL;DR

Sparse-ProxSkip is a novel method that combines sparse training and acceleration techniques to efficiently address resource and communication challenges in federated learning, supported by theoretical analysis and extensive experiments.

Contribution

It introduces Sparse-ProxSkip, integrating Straight-Through Estimator pruning with acceleration in sparse federated learning, overcoming naive combination issues.

Findings

Sparse-ProxSkip improves communication efficiency in FL.

Theoretical analysis explains why naive sparse-acceleration integration fails.

Extensive experiments validate the effectiveness of Sparse-ProxSkip.

Abstract

In Federated Learning (FL), both client resource constraints and communication costs pose major problems for training large models. In the centralized setting, sparse training addresses resource constraints, while in the distributed setting, local training addresses communication costs. Recent work has shown that local training provably improves communication complexity through acceleration. In this work we show that in FL, naive integration of sparse training and acceleration fails, and we provide theoretical and empirical explanations of this phenomenon. We introduce Sparse-ProxSkip, addressing the issue and implementing the efficient technique of Straight-Through Estimator pruning into sparse training. We demonstrate the performance of Sparse-ProxSkip in extensive experiments.