GHN-QAT: Training Graph Hypernetworks to Predict Quantization-Robust Parameters of Unseen Limited Precision Neural Networks

TL;DR

This paper investigates how quantization-aware training enhances the ability of Graph Hypernetworks to predict robust parameters for low-precision CNNs, improving accuracy especially at very low bitwidths.

Contribution

It demonstrates that quantization-aware training significantly boosts the quantized robustness of GHN-predicted parameters for 4-bit and 2-bit CNNs, advancing low-precision neural network deployment.

Findings

Quantization-aware training improves 4-bit CNN accuracy predicted by GHNs.

GHNs can achieve above-random accuracy for 2-bit CNNs with quantization-aware training.

Results suggest potential for using GHN-predicted parameters as initialization for quantized training.

Abstract

Graph Hypernetworks (GHN) can predict the parameters of varying unseen CNN architectures with surprisingly good accuracy at a fraction of the cost of iterative optimization. Following these successes, preliminary research has explored the use of GHNs to predict quantization-robust parameters for 8-bit and 4-bit quantized CNNs. However, this early work leveraged full-precision float32 training and only quantized for testing. We explore the impact of quantization-aware training and/or other quantization-based training strategies on quantized robustness and performance of GHN predicted parameters for low-precision CNNs. We show that quantization-aware training can significantly improve quantized accuracy for GHN predicted parameters of 4-bit quantized CNNs and even lead to greater-than-random accuracy for 2-bit quantized CNNs. These promising results open the door for future explorations such as investigating the use of GHN predicted parameters as initialization for further quantized training of individual CNNs, further exploration of "extreme bitwidth" quantization, and mixed precision quantization schemes.