Hardware Architecture of Embedded Inference Accelerator and Analysis of Algorithms for Depthwise and Large-Kernel Convolutions

TL;DR

This paper presents a flexible hardware architecture for CNN inference accelerators that efficiently supports depthwise and large-kernel convolutions, improving speed and reducing computational costs for embedded vision tasks.

Contribution

The proposed hardware architecture uniquely supports variable filter sizes without extra costs and introduces the DDC layer combining depthwise and dilated convolutions for better performance.

Findings

30% reduction in computational costs for face detection

20% decrease in model size with DDC layers

1% accuracy increase in image classification

Abstract

In order to handle modern convolutional neural networks (CNNs) efficiently, a hardware architecture of CNN inference accelerator is proposed to handle depthwise convolutions and regular convolutions, which are both essential building blocks for embedded-computer-vision algorithms. Different from related works, the proposed architecture can support filter kernels with different sizes with high flexibility since it does not require extra costs for intra-kernel parallelism, and it can generate convolution results faster than the architecture of the related works. The experimental results show the importance of supporting depthwise convolutions and dilated convolutions with the proposed hardware architecture. In addition to depthwise convolutions with large-kernels, a new structure called DDC layer, which includes the combination of depthwise convolutions and dilated convolutions, is also analyzed in this paper. For face detection, the computational costs decrease by 30%, and the model size decreases by 20% when the DDC layers are applied to the network. For image classification, the accuracy is increased by 1% by simply replacing $3 \times 3$ filters with $5 \times 5$ filters in depthwise convolutions.