Implementation of a framework for deploying AI inference engines in FPGAs

TL;DR

This paper presents a software framework that enables efficient deployment of machine learning inference engines on FPGAs, optimizing data flow and latency for real-time experimental applications at LCLS2.

Contribution

It introduces a novel FPGA deployment framework with a Keras-like API, supporting full redeployment and quantization, tailored for high-throughput, low-latency scientific data processing.

Findings

Preliminary framework successfully deploys ML models on FPGAs.

Optimized data streaming reduces latency and buffer requirements.

Supports dynamic network reconfiguration without resynthesis.

Abstract

The LCLS2 Free Electron Laser FEL will generate xray pulses to beamline experiments at up to 1Mhz These experimentals will require new ultrahigh rate UHR detectors that can operate at rates above 100 kHz and generate data throughputs upwards of 1 TBs a data velocity which requires prohibitively large investments in storage infrastructure Machine Learning has demonstrated the potential to digest large datasets to extract relevant insights however current implementations show latencies that are too high for realtime data reduction objectives SLAC has endeavored on the creation of a software framework which translates MLs structures for deployment on Field Programmable Gate Arrays FPGAs deployed at the Edge of the data chain close to the instrumentation This framework leverages Xilinxs HLS framework presenting an API modeled after the open source Keras interface to the TensorFlow library This SLAC Neural Network Library SNL framework is designed with a streaming data approach optimizing the data flow between layers while minimizing the buffer data buffering requirements The goal is to ensure the highest possible framerate while keeping the maximum latency constrained to the needs of the experiment Our framework is designed to ensure the RTL implementation of the network layers supporting full redeployment of weights and biases without requiring resynthesis after training The ability to reduce the precision of the implemented networks through quantization is necessary to optimize the use of both DSP and memory resources in the FPGA We currently have a preliminary version of the toolset and are experimenting with both general purpose example networks and networks being designed for specific LCLS2 experiments.