A Pre-defined Sparse Kernel Based Convolution for Deep CNNs

TL;DR

This paper introduces pSConv, a pre-defined sparse kernel convolution method that significantly reduces CNN complexity with minimal accuracy loss, demonstrated on CIFAR-10 and Tiny ImageNet datasets with ResNet18 and VGG16 architectures.

Contribution

The paper proposes pSConv, a novel sparse convolution technique that improves the complexity-accuracy trade-off in deep CNNs for training and inference.

Findings

Parameter count reduced by up to 4.24x

Outperforms ShuffleNet variant in efficiency

Achieves ~4% accuracy increase with fewer parameters

Abstract

The high demand for computational and storage resources severely impede the deployment of deep convolutional neural networks (CNNs) in limited-resource devices. Recent CNN architectures have proposed reduced complexity versions (e.g. SuffleNet and MobileNet) but at the cost of modest decreases inaccuracy. This paper proposes pSConv, a pre-defined sparse 2D kernel-based convolution, which promises significant improvements in the trade-off between complexity and accuracy for both CNN training and inference. To explore the potential of this approach, we have experimented with two widely accepted datasets, CIFAR-10 and Tiny ImageNet, in sparse variants of both the ResNet18 and VGG16 architectures. Our approach shows a parameter count reduction of up to 4.24x with modest degradation in classification accuracy relative to that of standard CNNs. Our approach outperforms a popular variant of ShuffleNet using a variant of ResNet18 with pSConv having 3x3 kernels with only four of nine elements not fixed at zero. In particular, the parameter count is reduced by 1.7x for CIFAR-10 and 2.29x for Tiny ImageNet with an increased accuracy of ~4%.