Fast convolutional neural networks on FPGAs with hls4ml

TL;DR

This paper presents an extension of the hls4ml library enabling deployment of ultra low-latency, low-power convolutional neural networks on FPGAs, with significant resource savings and minimal accuracy loss.

Contribution

We extend hls4ml to support convolutional neural networks on FPGAs, achieving microsecond latency and high resource efficiency through model compression techniques.

Findings

Achieved 5 microsecond inference latency on FPGA.

Reduced FPGA resource usage by up to 97% with no accuracy loss.

Demonstrated effective model compression methods like pruning and quantization.

Abstract

We introduce an automated tool for deploying ultra low-latency, low-power deep neural networks with convolutional layers on FPGAs. By extending the hls4ml library, we demonstrate an inference latency of $5\,\mu$s using convolutional architectures, targeting microsecond latency applications like those at the CERN Large Hadron Collider. Considering benchmark models trained on the Street View House Numbers Dataset, we demonstrate various methods for model compression in order to fit the computational constraints of a typical FPGA device used in trigger and data acquisition systems of particle detectors. In particular, we discuss pruning and quantization-aware training, and demonstrate how resource utilization can be significantly reduced with little to no loss in model accuracy. We show that the FPGA critical resource consumption can be reduced by 97% with zero loss in model accuracy, and by 99% when tolerating a 6% accuracy degradation.