Bitwidth-Adaptive Quantization-Aware Neural Network Training: A Meta-Learning Approach

TL;DR

This paper introduces MEBQAT, a meta-learning based method for neural network quantization that adapts to various bitwidths and target classes, enabling efficient deployment with minimal accuracy loss.

Contribution

It proposes a novel meta-learning approach for bitwidth-adaptive quantization-aware training, supporting both adaptive quantization and few-shot adaptation to unseen classes.

Findings

MEBQAT outperforms existing QAT methods in robustness and accuracy.

Supports joint adaptation of bitwidths and target classes.

Effective across multiple quantization schemes.

Abstract

Deep neural network quantization with adaptive bitwidths has gained increasing attention due to the ease of model deployment on various platforms with different resource budgets. In this paper, we propose a meta-learning approach to achieve this goal. Specifically, we propose MEBQAT, a simple yet effective way of bitwidth-adaptive quantization aware training (QAT) where meta-learning is effectively combined with QAT by redefining meta-learning tasks to incorporate bitwidths. After being deployed on a platform, MEBQAT allows the (meta-)trained model to be quantized to any candidate bitwidth then helps to conduct inference without much accuracy drop from quantization. Moreover, with a few-shot learning scenario, MEBQAT can also adapt a model to any bitwidth as well as any unseen target classes by adding conventional optimization or metric-based meta-learning. We design variants of MEBQAT to support both (1) a bitwidth-adaptive quantization scenario and (2) a new few-shot learning scenario where both quantization bitwidths and target classes are jointly adapted. We experimentally demonstrate their validity in multiple QAT schemes. By comparing their performance to (bitwidth-dedicated) QAT, existing bitwidth adaptive QAT and vanilla meta-learning, we find that merging bitwidths into meta-learning tasks achieves a higher level of robustness.