Free Bits: Latency Optimization of Mixed-Precision Quantized Neural Networks on the Edge

TL;DR

This paper introduces a hybrid search method for mixed-precision neural network quantization, optimizing latency on edge hardware with minimal accuracy loss, demonstrated on RISC-V microcontrollers with significant latency reductions.

Contribution

It proposes a hardware-aware differentiable and heuristic hybrid search algorithm for latency-optimized mixed-precision quantization tailored to specific hardware.

Findings

Up to 28.6% latency reduction on MobileNets with negligible accuracy loss.

Effective on diverse RISC-V microcontroller platforms.

Outperforms binary operation count-based search methods.

Abstract

Mixed-precision quantization, where a deep neural network's layers are quantized to different precisions, offers the opportunity to optimize the trade-offs between model size, latency, and statistical accuracy beyond what can be achieved with homogeneous-bit-width quantization. To navigate the intractable search space of mixed-precision configurations for a given network, this paper proposes a hybrid search methodology. It consists of a hardware-agnostic differentiable search algorithm followed by a hardware-aware heuristic optimization to find mixed-precision configurations latency-optimized for a specific hardware target. We evaluate our algorithm on MobileNetV1 and MobileNetV2 and deploy the resulting networks on a family of multi-core RISC-V microcontroller platforms with different hardware characteristics. We achieve up to 28.6% reduction of end-to-end latency compared to an 8-bit model at a negligible accuracy drop from a full-precision baseline on the 1000-class ImageNet dataset. We demonstrate speedups relative to an 8-bit baseline, even on systems with no hardware support for sub-byte arithmetic at negligible accuracy drop. Furthermore, we show the superiority of our approach with respect to differentiable search targeting reduced binary operation counts as a proxy for latency.