Accelerating Depthwise Separable Convolutions on Ultra-Low-Power Devices

TL;DR

This paper presents optimized methods for implementing depthwise separable convolutions on ultra-low-power devices, significantly reducing latency and data movement to enable efficient neural network deployment.

Contribution

It introduces novel kernel fusion and data layout strategies tailored for ultra-low-power hardware, achieving substantial performance improvements.

Findings

Up to 11.40% latency reduction on GAP8 SoC

Activation data movements reduced by up to 52.97%

Enhanced efficiency of depthwise separable convolutions on low-power devices

Abstract

Depthwise separable convolutions are a fundamental component in efficient Deep Neural Networks, as they reduce the number of parameters and operations compared to traditional convolutions while maintaining comparable accuracy. However, their low data reuse opportunities make deploying them notoriously difficult. In this work, we perform an extensive exploration of alternatives to fuse the depthwise and pointwise kernels that constitute the separable convolutional block. Our approach aims to minimize time-consuming memory transfers by combining different data layouts. When targeting a commercial ultra-low-power device with a three-level memory hierarchy, the GreenWaves GAP8 SoC, we reduce the latency of end-to-end network execution by up to 11.40%. Furthermore, our kernels reduce activation data movements between L2 and L1 memories by up to 52.97%.