FINN-GL: Generalized Mixed-Precision Extensions for FPGA-Accelerated LSTMs

TL;DR

This paper introduces a generalized FPGA-based deployment framework for LSTMs, enabling mixed-precision quantization and efficient hardware implementation, thus facilitating resource-efficient real-time RNN inference.

Contribution

It extends the FINN framework to support LSTMs using ONNX Scan operator, enabling mixed-precision quantization and hardware mapping for FPGA acceleration.

Findings

Achieves a balance between latency and resource use in FPGA LSTM accelerators.

Maintains or improves inference accuracy with reduced precision.

Demonstrates effectiveness on stock prediction task with FPGA implementation.

Abstract

Recurrent neural networks (RNNs), particularly LSTMs, are effective for time-series tasks like sentiment analysis and short-term stock prediction. However, their computational complexity poses challenges for real-time deployment in resource constrained environments. While FPGAs offer a promising platform for energy-efficient AI acceleration, existing tools mainly target feed-forward networks, and LSTM acceleration typically requires full custom implementation. In this paper, we address this gap by leveraging the open-source and extensible FINN framework to enable the generalized deployment of LSTMs on FPGAs. Specifically, we leverage the Scan operator from the Open Neural Network Exchange (ONNX) specification to model the recurrent nature of LSTM computations, enabling support for mixed quantisation within them and functional verification of LSTM-based models. Furthermore, we introduce custom transformations within the FINN compiler to map the quantised ONNX computation graph to hardware blocks from the HLS kernel library of the FINN compiler and Vitis HLS. We validate the proposed tool-flow by training a quantised ConvLSTM model for a mid-price stock prediction task using the widely used dataset and generating a corresponding hardware IP of the model using our flow, targeting the XCZU7EV device. We show that the generated quantised ConvLSTM accelerator through our flow achieves a balance between performance (latency) and resource consumption, while matching (or bettering) inference accuracy of state-of-the-art models with reduced precision. We believe that the generalisable nature of the proposed flow will pave the way for resource-efficient RNN accelerator designs on FPGAs.