Arithmetic-Intensity-Aware Quantization

TL;DR

AIQ is a post-training quantization framework that optimizes per-layer bit-widths to enhance arithmetic intensity and throughput in memory-bound neural network inference, with minimal accuracy loss.

Contribution

AIQ introduces a novel mixed precision quantization method that maximizes arithmetic intensity by selecting optimal per-layer bit-widths using search algorithms.

Findings

Increases AI by ~50% on ResNet-20/CIFAR-10 with minimal accuracy loss.

Achieves 1.66x higher throughput on MobileNetV2 with minimal accuracy drop.

Naturally quantizes larger layers more aggressively.

Abstract

As modern neural networks become increasingly memory-bound, inference throughput is limited by DRAM bandwidth rather than compute. We present Arithmetic-Intensity-Aware Quantization (AIQ), a mixed precision quantization framework that chooses per-layer bit-widths to maximize arithmetic intensity (AI) while minimizing accuracy loss. AIQ is a post-training quantization method that uses search algorithms over per-layer quantization schemes to minimize a weighted loss over AI and accuracy. On ResNet-20/CIFAR-10, AIQ increases AI by ~50% over an FP32 baseline while keeping test accuracy within ~1 percentage point, and outperforming global uniform quantization schemes. On a memory-bound MobileNetV2 architecture, AIQ configurations give a 1.66x higher throughput than the FP32 baseline while keeping test accuracy within 1 percentage point. We also find that AIQ naturally quantizes larger layers more aggressively.