Drastically Reducing the Number of Trainable Parameters in Deep CNNs by Inter-layer Kernel-sharing

TL;DR

This paper proposes a simple kernel-sharing method between isomorphic layers in deep CNNs to drastically reduce trainable parameters, enabling efficient edge computing with minimal accuracy loss.

Contribution

Introducing kernel-sharing between isomorphic layers in CNNs as a novel way to reduce parameters and improve regularization.

Findings

Significant parameter reduction with minimal accuracy loss

Effective regularization reducing overfitting

Suitable for memory-constrained edge devices

Abstract

Deep convolutional neural networks (DCNNs) have become the state-of-the-art (SOTA) approach for many computer vision tasks: image classification, object detection, semantic segmentation, etc. However, most SOTA networks are too large for edge computing. Here, we suggest a simple way to reduce the number of trainable parameters and thus the memory footprint: sharing kernels between multiple convolutional layers. Kernel-sharing is only possible between ``isomorphic" layers, i.e.layers having the same kernel size, input and output channels. This is typically the case inside each stage of a DCNN. Our experiments on CIFAR-10 and CIFAR-100, using the ConvMixer and SE-ResNet architectures show that the number of parameters of these models can drastically be reduced with minimal cost on accuracy. The resulting networks are appealing for certain edge computing applications that are subject to severe memory constraints, and even more interesting if leveraging "frozen weights" hardware accelerators. Kernel-sharing is also an efficient regularization method, which can reduce overfitting. The codes are publicly available at https://github.com/AlirezaAzadbakht/kernel-sharing.