Learning Efficient Convolutional Networks through Irregular Convolutional Kernels

TL;DR

This paper introduces a novel method for reducing neural network parameters by transforming convolutional kernels into irregular line segments, significantly decreasing model size and computation with minimal accuracy loss.

Contribution

It proposes a new kernel structure transformation and learning strategy that effectively prunes parameters and computations in deep networks for low-power devices.

Findings

Reduced parameters by 69% on DenseNet-40

Decreased calculations by 59% on DenseNet-40

Maintained accuracy within 2% of original performance

Abstract

As deep neural networks are increasingly used in applications suited for low-power devices, a fundamental dilemma becomes apparent: the trend is to grow models to absorb increasing data that gives rise to memory intensive; however low-power devices are designed with very limited memory that can not store large models. Parameters pruning is critical for deep model deployment on low-power devices. Existing efforts mainly focus on designing highly efficient structures or pruning redundant connections for networks. They are usually sensitive to the tasks or relay on dedicated and expensive hashing storage strategies. In this work, we introduce a novel approach for achieving a lightweight model from the views of reconstructing the structure of convolutional kernels and efficient storage. Our approach transforms a traditional square convolution kernel to line segments, and automatically learn a proper strategy for equipping these line segments to model diverse features. The experimental results indicate that our approach can massively reduce the number of parameters (pruned 69% on DenseNet-40) and calculations (pruned 59% on DenseNet-40) while maintaining acceptable performance (only lose less than 2% accuracy).