FedDIP: Federated Learning with Extreme Dynamic Pruning and Incremental Regularization

TL;DR

FedDIP introduces a federated learning framework that combines dynamic pruning and incremental regularization to reduce communication costs while maintaining high accuracy in training large DNNs.

Contribution

It proposes a novel FL method integrating adaptive pruning with regularization, enabling extreme sparsity and efficient training without sacrificing performance.

Findings

Achieves high model sparsity with maintained accuracy.

Outperforms existing pruning methods in federated settings.

Provides convergence analysis and comprehensive benchmarks.

Abstract

Federated Learning (FL) has been successfully adopted for distributed training and inference of large-scale Deep Neural Networks (DNNs). However, DNNs are characterized by an extremely large number of parameters, thus, yielding significant challenges in exchanging these parameters among distributed nodes and managing the memory. Although recent DNN compression methods (e.g., sparsification, pruning) tackle such challenges, they do not holistically consider an adaptively controlled reduction of parameter exchange while maintaining high accuracy levels. We, therefore, contribute with a novel FL framework (coined FedDIP), which combines (i) dynamic model pruning with error feedback to eliminate redundant information exchange, which contributes to significant performance improvement, with (ii) incremental regularization that can achieve \textit{extreme} sparsity of models. We provide convergence analysis of FedDIP and report on a comprehensive performance and comparative assessment against state-of-the-art methods using benchmark data sets and DNN models. Our results showcase that FedDIP not only controls the model sparsity but efficiently achieves similar or better performance compared to other model pruning methods adopting incremental regularization during distributed model training. The code is available at: https://github.com/EricLoong/feddip.