Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods

TL;DR

This paper introduces an iterative hard thresholding method to train Skinny Deep Neural Networks with fewer parameters, achieving competitive performance and improved efficiency on multiple benchmark datasets.

Contribution

It proposes a novel IHT approach for training SDNNs, enabling effective parameter reduction while maintaining or improving accuracy.

Findings

SDNNs trained with IHT outperform full models in efficiency.

IHT-based SDNNs achieve comparable or better accuracy on benchmarks.

The method is applicable to various CNN architectures like NIN and AlexNet.

Abstract

Deep neural networks have achieved remarkable success in a wide range of practical problems. However, due to the inherent large parameter space, deep models are notoriously prone to overfitting and difficult to be deployed in portable devices with limited memory. In this paper, we propose an iterative hard thresholding (IHT) approach to train Skinny Deep Neural Networks (SDNNs). An SDNN has much fewer parameters yet can achieve competitive or even better performance than its full CNN counterpart. More concretely, the IHT approach trains an SDNN through following two alternative phases: (I) perform hard thresholding to drop connections with small activations and fine-tune the other significant filters; (II)~re-activate the frozen connections and train the entire network to improve its overall discriminative capability. We verify the superiority of SDNNs in terms of efficiency and classification performance on four benchmark object recognition datasets, including CIFAR-10, CIFAR-100, MNIST and ImageNet. Experimental results clearly demonstrate that IHT can be applied for training SDNN based on various CNN architectures such as NIN and AlexNet.