A New Clustering-Based Technique for the Acceleration of Deep Convolutional Networks

TL;DR

This paper introduces a clustering-based model compression technique for deep neural networks that enhances acceleration and efficiency, especially suitable for resource-limited devices, by structuring representatives to outperform traditional methods.

Contribution

The paper proposes a novel clustering approach with a structured representative design that improves acceleration gains over standard k-means in model compression.

Findings

The method achieves higher acceleration compared to conventional k-means.

Extensive evaluations validate the method's superiority across various DNN models.

Theoretical analysis confirms the potential for increased efficiency in model compression.

Abstract

Deep learning and especially the use of Deep Neural Networks (DNNs) provides impressive results in various regression and classification tasks. However, to achieve these results, there is a high demand for computing and storing resources. This becomes problematic when, for instance, real-time, mobile applications are considered, in which the involved (embedded) devices have limited resources. A common way of addressing this problem is to transform the original large pre-trained networks into new smaller models, by utilizing Model Compression and Acceleration (MCA) techniques. Within the MCA framework, we propose a clustering-based approach that is able to increase the number of employed centroids/representatives, while at the same time, have an acceleration gain compared to conventional, $k$-means based approaches. This is achieved by imposing a special structure to the employed representatives, which is enabled by the particularities of the problem at hand. Moreover, the theoretical acceleration gains are presented and the key system hyper-parameters that affect that gain, are identified. Extensive evaluation studies carried out using various state-of-the-art DNN models trained in image classification, validate the superiority of the proposed method as compared for its use in MCA tasks.