Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey

TL;DR

This survey reviews methods that reduce training costs of deep neural networks by pruning and freezing parts of the model, introducing a unified framework called dimensionality reduced training.

Contribution

It introduces the concept of dimensionality reduced training as a unifying mathematical model for pruning and freezing techniques during training.

Findings

Pruning methods include at initialization, lottery tickets, and dynamic sparse training.

Freezing weights reduces trainable parameters and computational costs.

The survey categorizes and discusses various dimensionality reduction techniques during training.

Abstract

State-of-the-art deep learning models have a parameter count that reaches into the billions. Training, storing and transferring such models is energy and time consuming, thus costly. A big part of these costs is caused by training the network. Model compression lowers storage and transfer costs, and can further make training more efficient by decreasing the number of computations in the forward and/or backward pass. Thus, compressing networks also at training time while maintaining a high performance is an important research topic. This work is a survey on methods which reduce the number of trained weights in deep learning models throughout the training. Most of the introduced methods set network parameters to zero which is called pruning. The presented pruning approaches are categorized into pruning at initialization, lottery tickets and dynamic sparse training. Moreover, we discuss methods that freeze parts of a network at its random initialization. By freezing weights, the number of trainable parameters is shrunken which reduces gradient computations and the dimensionality of the model's optimization space. In this survey we first propose dimensionality reduced training as an underlying mathematical model that covers pruning and freezing during training. Afterwards, we present and discuss different dimensionality reduced training methods.