Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization

TL;DR

This paper introduces a novel dynamic sparse reparameterization technique for training deep convolutional neural networks efficiently, outperforming previous methods by dynamically reallocating parameters during training, leading to better accuracy with fewer parameters.

Contribution

The paper proposes a new dynamic sparse reparameterization method that reduces computational costs and manual tuning, improving training efficiency and accuracy of sparse neural networks.

Findings

Our method achieves higher accuracy than previous static and dynamic reparameterization techniques.

Dynamic exploration of network structure during training is crucial for superior generalization.

Sparse networks trained with our method match the accuracy of pruned dense models.

Abstract

Modern deep neural networks are typically highly overparameterized. Pruning techniques are able to remove a significant fraction of network parameters with little loss in accuracy. Recently, techniques based on dynamic reallocation of non-zero parameters have emerged, allowing direct training of sparse networks without having to pre-train a large dense model. Here we present a novel dynamic sparse reparameterization method that addresses the limitations of previous techniques such as high computational cost and the need for manual configuration of the number of free parameters allocated to each layer. We evaluate the performance of dynamic reallocation methods in training deep convolutional networks and show that our method outperforms previous static and dynamic reparameterization methods, yielding the best accuracy for a fixed parameter budget, on par with accuracies obtained by iteratively pruning a pre-trained dense model. We further investigated the mechanisms underlying the superior generalization performance of the resultant sparse networks. We found that neither the structure, nor the initialization of the non-zero parameters were sufficient to explain the superior performance. Rather, effective learning crucially depended on the continuous exploration of the sparse network structure space during training. Our work suggests that exploring structural degrees of freedom during training is more effective than adding extra parameters to the network.