A Deeper Look into Convolutions via Eigenvalue-based Pruning

TL;DR

This paper investigates eigenvalue-based pruning of convolutional kernels in ResNets, revealing how many kernels can be removed without affecting classification accuracy, thereby providing insights into CNN internal mechanisms.

Contribution

It introduces an eigenvalue-based method for pruning CNN kernels, offering a new perspective on kernel importance beyond traditional weight-based metrics.

Findings

Eigenvalue-based pruning preserves accuracy with fewer kernels.

Many convolutional kernels can be eliminated without performance loss.

Provides insights into CNN internal structure and kernel importance.

Abstract

Convolutional neural networks (CNNs) are able to attain better visual recognition performance than fully connected neural networks despite having much fewer parameters due to their parameter sharing principle. Modern architectures usually contain a small number of fully-connected layers, often at the end, after multiple layers of convolutions. In some cases, most of the convolutions can be eliminated without suffering any loss in recognition performance. However, there is no solid recipe to detect the hidden subset of convolutional neurons that is responsible for the majority of the recognition work. In this work, we formulate this as a pruning problem where the aim is to prune as many kernels as possible while preserving the vanilla generalization performance. To this end, we use the matrix characteristics based on eigenvalues for pruning, in comparison to the average absolute weight of a kernel which is the de facto standard in the literature to assess the importance of an individual convolutional kernel, to shed light on the internal mechanisms of a widely used family of CNNs, namely residual neural networks (ResNets), for the image classification problem using CIFAR-10, CIFAR-100 and Tiny ImageNet datasets.