MCU-MixQ: A HW/SW Co-optimized Mixed-precision Neural Network Design Framework for MCUs

TL;DR

MCU-MixQ is a hardware/software co-designed framework that optimizes mixed-precision neural networks for microcontrollers by leveraging SIMD packing and neural architecture search, significantly improving processing speed under resource constraints.

Contribution

The paper introduces a novel SIMD packing technique and a co-optimized NAS-based framework for efficient mixed-precision neural network deployment on MCUs.

Findings

Achieves 2.1× speedup over CMix-NN

Achieves 1.4× speedup over MCUNet

Effectively balances neural network accuracy and performance

Abstract

Mixed-precision neural network (MPNN) that utilizes just enough data width for the neural network processing is an effective approach to meet the stringent resources constraints including memory and computing of MCUs. Nevertheless, there is still a lack of sub-byte and mixed-precision SIMD operations in MCU-class ISA and the limited computing capability of MCUs remains underutilized, which further aggravates the computing bound encountered in neural network processing. As a result, the benefits of MPNNs cannot be fully unleashed. In this work, we propose to pack multiple low-bitwidth arithmetic operations within a single instruction multiple data (SIMD) instructions in typical MCUs, and then develop an efficient convolution operator by exploring both the data parallelism and computing parallelism in convolution along with the proposed SIMD packing. Finally, we further leverage Neural Architecture Search (NAS) to build a HW/SW co-designed MPNN design framework, namely MCU-MixQ. This framework can optimize both the MPNN quantization and MPNN implementation efficiency, striking an optimized balance between neural network performance and accuracy. According to our experiment results, MCU-MixQ achieves 2.1$\times$ and 1.4$\times$ speedup over CMix-NN and MCUNet respectively under the same resource constraints.