Towards Sparsified Federated Neuroimaging Models via Weight Pruning

TL;DR

This paper introduces FedSparsify, a method for pruning neural networks during federated training, achieving up to 95% sparsity without performance loss and enhancing privacy against inference attacks.

Contribution

The paper presents a novel federated training approach that integrates progressive model pruning, significantly reducing communication costs and improving privacy in federated neuroimaging models.

Findings

Models can be pruned up to 95% sparsity without performance loss.

High sparsity models are less vulnerable to membership inference attacks.

Pruning during federated training reduces communication and training costs.

Abstract

Federated training of large deep neural networks can often be restrictive due to the increasing costs of communicating the updates with increasing model sizes. Various model pruning techniques have been designed in centralized settings to reduce inference times. Combining centralized pruning techniques with federated training seems intuitive for reducing communication costs -- by pruning the model parameters right before the communication step. Moreover, such a progressive model pruning approach during training can also reduce training times/costs. To this end, we propose FedSparsify, which performs model pruning during federated training. In our experiments in centralized and federated settings on the brain age prediction task (estimating a person's age from their brain MRI), we demonstrate that models can be pruned up to 95% sparsity without affecting performance even in challenging federated learning environments with highly heterogeneous data distributions. One surprising benefit of model pruning is improved model privacy. We demonstrate that models with high sparsity are less susceptible to membership inference attacks, a type of privacy attack.