De-specializing an HLS library for Deep Neural Networks: improvements upon hls4ml

TL;DR

This paper analyzes and improves the hls4ml framework to enhance FPGA-based neural network acceleration, enabling more advanced optimizations, broader FPGA targeting, and larger model support.

Contribution

It proposes enhancements to the hls4ml library, allowing for more advanced optimizations and wider FPGA and model support.

Findings

Improved core library components for hls4ml.

Enhanced optimization capabilities for neural network deployment.

Broader FPGA compatibility and larger model support.

Abstract

Custom hardware accelerators for Deep Neural Networks are increasingly popular: in fact, the flexibility and performance offered by FPGAs are well-suited to the computational effort and low latency constraints required by many image recognition and natural language processing tasks. The gap between high-level Machine Learning frameworks (e.g., Tensorflow, Pytorch) and low-level hardware design in Verilog/VHDL creates a barrier to widespread adoption of FPGAs, which can be overcome with the help of High-Level Synthesis. hls4ml is a framework that translates Deep Neural Networks into annotated C++ code for High-Level Synthesis, offering a complete and user-friendly design process that has been enthusiastically adopted in physics research. We analyze the strengths and weaknesses of hls4ml, drafting a plan to enhance its core library of components in order to allow more advanced optimizations, target a wider selection of FPGAs, and support larger Neural Network models.