DSXplore: Optimizing Convolutional Neural Networks via Sliding-Channel Convolutions

TL;DR

DSXplore introduces a novel sliding-channel convolution (SCC) technique to optimize depthwise separable convolutions in CNNs, achieving better accuracy and efficiency for mobile and resource-constrained applications.

Contribution

The paper presents the first optimized design for exploring DSCs using SCC, including an algorithmic approach and GPU implementation, surpassing existing methods in accuracy and efficiency.

Findings

DSXplore outperforms standard convolution and existing DSCs in accuracy and efficiency.

SCC provides adjustable parameters for balancing accuracy and computational cost.

GPU implementation of SCC achieves significant speedups on various datasets.

Abstract

As the key advancement of the convolutional neural networks (CNNs), depthwise separable convolutions (DSCs) are becoming one of the most popular techniques to reduce the computations and parameters size of CNNs meanwhile maintaining the model accuracy. It also brings profound impact to improve the applicability of the compute- and memory-intensive CNNs to a broad range of applications, such as mobile devices, which are generally short of computation power and memory. However, previous research in DSCs are largely focusing on compositing the limited existing DSC designs, thus, missing the opportunities to explore more potential designs that can achieve better accuracy and higher computation/parameter reduction. Besides, the off-the-shelf convolution implementations offer limited computing schemes, therefore, lacking support for DSCs with different convolution patterns. To this end, we introduce, DSXplore, the first optimized design for exploring DSCs on CNNs. Specifically, at the algorithm level, DSXplore incorporates a novel factorized kernel -- sliding-channel convolution (SCC), featured with input-channel overlapping to balance the accuracy performance and the reduction of computation and memory cost. SCC also offers enormous space for design exploration by introducing adjustable kernel parameters. Further, at the implementation level, we carry out an optimized GPU-implementation tailored for SCC by leveraging several key techniques, such as the input-centric backward design and the channel-cyclic optimization. Intensive experiments on different datasets across mainstream CNNs show the advantages of DSXplore in balancing accuracy and computation/parameter reduction over the standard convolution and the existing DSCs.