OrthoReg: Robust Network Pruning Using Orthonormality Regularization

TL;DR

OrthoReg introduces an orthonormality regularization technique for CNN filters, reducing correlation and improving the accuracy and efficiency of network pruning, especially in early training stages.

Contribution

This paper presents a novel regularization strategy that enforces orthonormality on CNN filters, enabling more reliable importance estimation and effective pruning of large filter groups.

Findings

Outperforms five baseline pruning techniques on CIFAR-100 and Tiny-ImageNet.

Enhances the Early-Bird Ticket hypothesis by enabling effective pruning early in training.

Improves trainability and accuracy of pruned networks.

Abstract

Network pruning in Convolutional Neural Networks (CNNs) has been extensively investigated in recent years. To determine the impact of pruning a group of filters on a network's accuracy, state-of-the-art pruning methods consistently assume filters of a CNN are independent. This allows the importance of a group of filters to be estimated as the sum of importances of individual filters. However, overparameterization in modern networks results in highly correlated filters that invalidate this assumption, thereby resulting in incorrect importance estimates. To address this issue, we propose OrthoReg, a principled regularization strategy that enforces orthonormality on a network's filters to reduce inter-filter correlation, thereby allowing reliable, efficient determination of group importance estimates, improved trainability of pruned networks, and efficient, simultaneous pruning of large groups of filters. When used for iterative pruning on VGG-13, MobileNet-V1, and ResNet-34, OrthoReg consistently outperforms five baseline techniques, including the state-of-the-art, on CIFAR-100 and Tiny-ImageNet. For the recently proposed Early-Bird Ticket hypothesis, which claims networks become amenable to pruning early-on in training and can be pruned after a few epochs to minimize training expenditure, we find OrthoReg significantly outperforms prior work. Code available at https://github.com/EkdeepSLubana/OrthoReg.