Modulating Regularization Frequency for Efficient Compression-Aware Model Training

TL;DR

This paper proposes a novel regularization technique called regularization frequency, which optimizes the timing of compression during training to improve the efficiency and accuracy of compression-aware neural network training.

Contribution

It introduces regularization frequency as a new parameter to control compression regularization strength, enhancing training efficiency and model performance.

Findings

Regularization frequency significantly impacts model accuracy.

Combining regularization frequency with compression ratio improves training outcomes.

Occasional compression can match or outperform frequent compression.

Abstract

While model compression is increasingly important because of large neural network size, compression-aware training is challenging as it needs sophisticated model modifications and longer training time.In this paper, we introduce regularization frequency (i.e., how often compression is performed during training) as a new regularization technique for a practical and efficient compression-aware training method. For various regularization techniques, such as weight decay and dropout, optimizing the regularization strength is crucial to improve generalization in Deep Neural Networks (DNNs). While model compression also demands the right amount of regularization, the regularization strength incurred by model compression has been controlled only by compression ratio. Throughout various experiments, we show that regularization frequency critically affects the regularization strength of model compression. Combining regularization frequency and compression ratio, the amount of weight updates by model compression per mini-batch can be optimized to achieve the best model accuracy. Modulating regularization frequency is implemented by occasional model compression while conventional compression-aware training is usually performed for every mini-batch.