HGQ: High Granularity Quantization for Real-time Neural Networks on FPGAs

TL;DR

HGQ is a novel quantization-aware training framework that assigns optimal bit-widths to neural network parameters independently, enabling real-time, low-latency inference on FPGAs for critical applications.

Contribution

HGQ introduces a gradient descent-based method for per-parameter bit-width optimization, supporting heterogeneous precision and significantly reducing resource use and latency.

Findings

Achieves orders of magnitude reduction in resource consumption

Maintains accuracy across multiple benchmark tasks

Enables deployment of complex models in latency-critical applications

Abstract

Neural networks with sub-microsecond inference latency are required by many critical applications. Targeting such applications deployed on FPGAs, we present High Granularity Quantization (HGQ), a quantization-aware training framework that optimizes parameter bit-widths through gradient descent. Unlike conventional methods, HGQ determines the optimal bit-width for each parameter independently, making it suitable for hardware platforms supporting heterogeneous arbitrary precision arithmetic. In our experiments, HGQ shows superior performance compared to existing network compression methods, achieving orders of magnitude reduction in resource consumption and latency while maintaining the accuracy on several benchmark tasks. These improvements enable the deployment of complex models previously infeasible due to resource or latency constraints. HGQ is open-source and is used for developing next-generation trigger systems at the CERN ATLAS and CMS experiments for particle physics, enabling the use of advanced machine learning models for real-time data selection with sub-microsecond latency.