PolyLUT: Ultra-low Latency Polynomial Inference with Hardware-Aware Structured Pruning

TL;DR

PolyLUT introduces a hardware-aware polynomial-based neural network inference method optimized for FPGA deployment, reducing latency and area by using structured pruning and polynomial evaluation within LUTs.

Contribution

The paper presents a novel training approach for DNNs using multivariate polynomials and a structured pruning strategy tailored for FPGA hardware, enabling ultra-low latency inference.

Findings

Achieves comparable accuracy with fewer layers using polynomial blocks.

Significant latency and area reductions demonstrated on multiple tasks.

Effective structured pruning improves hardware efficiency.

Abstract

Standard deep neural network inference involves the computation of interleaved linear maps and nonlinear activation functions. Prior work for ultra-low latency implementations has hardcoded these operations inside FPGA lookup tables (LUTs). However, FPGA LUTs can implement a much greater variety of functions. In this paper, we propose a novel approach to training DNNs for FPGA deployment using multivariate polynomials as the basic building block. Our method takes advantage of the flexibility offered by the soft logic, hiding the polynomial evaluation inside the LUTs with minimal overhead. By using polynomial building blocks, we achieve the same accuracy using considerably fewer layers of soft logic than by using linear functions, leading to significant latency and area improvements. LUT-based implementations also face a significant challenge: the LUT size grows exponentially with the number of inputs. Prior work relies on a priori fixed sparsity, with results heavily dependent on seed selection. To address this, we propose a structured pruning strategy using a bespoke hardware-aware group regularizer that encourages a particular sparsity pattern that leads to a small number of inputs per neuron. We demonstrate the effectiveness of PolyLUT on three tasks: network intrusion detection, jet identification at the CERN Large Hadron Collider, and MNIST.