Channel-wise Mixed-precision Assignment for DNN Inference on Constrained Edge Nodes

TL;DR

This paper introduces a novel neural architecture search method that assigns mixed-precision bit-widths independently to each weight tensor channel, significantly improving efficiency on edge devices while maintaining accuracy.

Contribution

It proposes a new NAS approach that independently assigns bit-widths per channel, expanding the search space beyond layer-wise methods for better efficiency.

Findings

Achieved up to 63% reduction in memory usage.

Reduced energy consumption by up to 27%.

Produced Pareto-optimal models balancing accuracy and efficiency.

Abstract

Quantization is widely employed in both cloud and edge systems to reduce the memory occupation, latency, and energy consumption of deep neural networks. In particular, mixed-precision quantization, i.e., the use of different bit-widths for different portions of the network, has been shown to provide excellent efficiency gains with limited accuracy drops, especially with optimized bit-width assignments determined by automated Neural Architecture Search (NAS) tools. State-of-the-art mixed-precision works layer-wise, i.e., it uses different bit-widths for the weights and activations tensors of each network layer. In this work, we widen the search space, proposing a novel NAS that selects the bit-width of each weight tensor channel independently. This gives the tool the additional flexibility of assigning a higher precision only to the weights associated with the most informative features. Testing on the MLPerf Tiny benchmark suite, we obtain a rich collection of Pareto-optimal models in the accuracy vs model size and accuracy vs energy spaces. When deployed on the MPIC RISC-V edge processor, our networks reduce the memory and energy for inference by up to 63% and 27% respectively compared to a layer-wise approach, for the same accuracy.