QBitOpt: Fast and Accurate Bitwidth Reallocation during Training

TL;DR

QBitOpt is a novel algorithm that efficiently allocates bitwidths during quantization-aware training, producing high-performance mixed-precision neural networks that meet strict resource constraints.

Contribution

It formulates bitwidth allocation as a constrained optimization problem and introduces a fast, sensitivity-based method that guarantees resource constraints during training.

Findings

Outperforms existing fixed and mixed-precision methods on ImageNet

Produces high-accuracy networks under strict bitwidth constraints

Guarantees resource constraints during training

Abstract

Quantizing neural networks is one of the most effective methods for achieving efficient inference on mobile and embedded devices. In particular, mixed precision quantized (MPQ) networks, whose layers can be quantized to different bitwidths, achieve better task performance for the same resource constraint compared to networks with homogeneous bitwidths. However, finding the optimal bitwidth allocation is a challenging problem as the search space grows exponentially with the number of layers in the network. In this paper, we propose QBitOpt, a novel algorithm for updating bitwidths during quantization-aware training (QAT). We formulate the bitwidth allocation problem as a constraint optimization problem. By combining fast-to-compute sensitivities with efficient solvers during QAT, QBitOpt can produce mixed-precision networks with high task performance guaranteed to satisfy strict resource constraints. This contrasts with existing mixed-precision methods that learn bitwidths using gradients and cannot provide such guarantees. We evaluate QBitOpt on ImageNet and confirm that we outperform existing fixed and mixed-precision methods under average bitwidth constraints commonly found in the literature.