Neural Network Acceleration on MPSoC board: Integrating SLAC's SNL, Rogue Software and Auto-SNL

TL;DR

This paper introduces SNL and Auto-SNL, frameworks for deploying real-time neural network inference on FPGAs, demonstrating competitive latency and resource efficiency compared to existing tools like hls4ml.

Contribution

The paper presents SNL, a flexible FPGA deployment framework with dynamic weight updates, and Auto-SNL, a Python tool for converting models into FPGA-compatible code, along with a benchmark comparison to hls4ml.

Findings

SNL achieves comparable or better latency than hls4ml.

SNL can reduce FPGA resource usage in some architectures.

Auto-SNL simplifies the conversion process from Python models.

Abstract

The LCLS-II Free Electron Laser (FEL) will generate X-ray pulses for beamline experiments at rates of up to 1~MHz, with detectors producing data throughputs exceeding 1 TB/s. Managing such massive data streams presents significant challenges, as transmission and storage infrastructures become prohibitively expensive. Machine learning (ML) offers a promising solution for real-time data reduction, but conventional implementations introduce excessive latency, making them unsuitable for high-speed experimental environments. To address these challenges, SLAC developed the SLAC Neural Network Library (SNL), a specialized framework designed to deploy real-time ML inference models on Field-Programmable Gate Arrays (FPGA). SNL's key feature is the ability to dynamically update model weights without requiring FPGA resynthesis, enhancing flexibility for adaptive learning applications. To further enhance usability and accessibility, we introduce Auto-SNL, a Python extension that streamlines the process of converting Python-based neural network models into SNL-compatible high-level synthesis code. This paper presents a benchmark comparison against hls4ml, the current state-of-the-art tool, across multiple neural network architectures, fixed-point precisions, and synthesis configurations targeting a Xilinx ZCU102 FPGA. The results showed that SNL achieves competitive or superior latency in most tested architectures, while in some cases also offering FPGA resource savings. This adaptation demonstrates SNL's versatility, opening new opportunities for researchers and academics in fields such as high-energy physics, medical imaging, robotics, and many more.

Tables

Table 1. TABLE I: Benchmark task, dataset, and performance summary.

Model	Dataset	Input size	Task	Performance
Jet	LHC Jet [20]	(16,)	Classification	74.90%
Anomaly	ToyADMOS [21]	(320,)	Detection	0.70 (AUC)
KWS	Speech Commands [22]	(32, 32, 1)	Classification	59.33%
VWW	Visual Wake Words [23]	(49, 10, 1)	Classification	70.14%

Table 2. TABLE II: Hardware synthesis parameter space.

	hls4ml	SNL
Precision	$⟨32,16⟩$, $⟨16,6⟩$, $⟨8,3⟩$	$⟨32,16⟩$, $⟨16,6⟩$, $⟨8,3⟩$
Strategy	Latency, Resource	N/A
Reuse factor	1, 2, 4, 8	N/A
IO Type	Stream	N/A
Clock period	$10\mathrm{ns}$	$10\mathrm{ns}$

Table 3. TABLE III: Training hyperparameters and network architectures.

Model	Batch size	Learning rate	Architecture
Jet	1024	10-4	$\underset{\text{Dense}}{\overset{\text{Neurons}=64}{\to }}\underset{\text{ReLU}}{\overset{}{\to }}\underset{\text{Dense}}{\overset{32}{\to }}\underset{\text{ReLU}}{\overset{}{\to }}\underset{\text{Dense}}{\overset{32}{\to }}\underset{\text{ReLU}}{\overset{}{\to }}\underset{\text{Dense}}{\overset{5}{\to }}\underset{\text{Softmax}}{\overset{}{\to }}$
Anomaly	512	10-4	$\underset{\text{Dense}}{\overset{16}{\to }}\underset{\text{ReLU}}{\overset{}{\to }}\underset{\text{Dense}}{\overset{32}{\to }}\underset{\text{ReLU}}{\overset{}{\to }}\underset{\text{Dense}}{\overset{32}{\to }}\underset{\text{ReLU}}{\overset{}{\to }}\underset{\text{Dense}}{\overset{8}{\to }}\underset{\text{ReLU}}{\overset{}{\to }}\underset{\text{Dense}}{\overset{32}{\to }}\underset{\text{ReLU}}{\overset{}{\to }}\underset{\text{Dense}}{\overset{32}{\to }}\underset{\text{ReLU}}{\overset{}{\to }}\underset{\text{Dense}}{\overset{16}{\to }}\underset{\text{ReLU}}{\overset{}{\to }}\underset{\text{Dense}}{\overset{320}{\to }}$
KWS	64	10-4	$\underset{\text{Conv2D}}{\overset{\text{Filters}=16\text{Kernel}=(5,5)}{\to }}\underset{\text{ReLU}}{\overset{}{\to }}\underset{\text{Conv2D}}{\overset{8(3,3)}{\to }}\underset{\text{ReLU}}{\overset{}{\to }}\underset{\text{Dropout}}{\overset{\text{Rate}=0.2}{\to }}\underset{\text{GlobalAveragePooling2D}}{\overset{}{\to }}\underset{\text{Dense}}{\overset{12}{\to }}\underset{\text{Softmax}}{\overset{}{\to }}$
VWW	32	10-4	$\underset{\text{Conv2D}}{\overset{4(3,3)}{\to }}\underset{\text{ReLU}}{\overset{}{\to }}\underset{\text{AveragePooling2D}}{\overset{\text{Pool}=(2,2)\text{Strides}=(2,2)}{\to }}\underset{\text{Conv2D}}{\overset{4(3,3)}{\to }}\underset{\text{ReLU}}{\overset{}{\to }}\underset{\text{GlobalAveragePooling2D}}{\overset{}{\to }}\underset{\text{Dense}}{\overset{2}{\to }}\underset{\text{Softmax}}{\overset{}{\to }}$

Full text

I Introduction

Modern ultra-high-rate (UHR) experimental facilities such as the Linac Coherent Light Source II (LCLS‑II) push the limits of scientific imaging by delivering X-ray pulses at repetition rates up to 1 MHz. These unprecedented rates allow for detailed time-resolved studies in material science, chemistry, and biology, but also generate data at a staggering rate of over 1 TB/s, far outpacing the capacity of conventional data storage and processing systems [1].

Machine learning (ML) has emerged as a powerful approach for real-time data reduction by moving inference to the edge [2, 3]. While CPUs and GPUs are common ML deployment platforms, they often fall short in meeting the ultra-low-latency demands of facilities like LCLS-II. In such environments, inference pipelines must operate within microsecond-scale latency budgets, making FPGAs at the edge a compelling choice for low-latency, high-throughput ML inference.

In response, SLAC developed the SLAC Neural Network Library (SNL) [1], a domain-specific HLS-based framework designed to deploy ML models on FPGAs for real-time experimental applications. A defining feature of SNL is its ability to dynamically reload neural network (NN) weights and biases without requiring FPGA resynthesis, thus supporting rapid iteration, retraining, and deployment in mission-critical environments [4]. To further democratize FPGA-based ML deployment, we introduce Auto-SNL, a Python extension that automates the conversion of Python-defined models into SNL-compatible high-level synthesis (HLS) code. Auto-SNL lowers the barrier to entry by abstracting away hardware intricacies, allowing non-expert users to generate optimized bitfiles and control hardware parameters without directly engaging with FPGA toolchains.

One of the main goals of this work is to position SNL’s current capabilities within the broader landscape of ML-to-FPGA frameworks. While SNL and Auto-SNL address the specific needs of SLAC’s ongoing experiments, it is important to understand how they compare with other available tools in terms of features, flexibility, design-space exploration, and synthesis efficiency.

Several FPGA-oriented ML deployment frameworks have also emerged in recent years. Vendor-supported toolchains such as Xilinx Vitis AI [5] and Intel OpenVINO [6] provide optimized pipelines for neural networks using pre-built and proprietary IP blocks. FINN [7] targets extreme quantization for ultra-low latency embedded inference, while other research projects explore domain-specific accelerators or specialize in a particular set of ML architectures [8, 9].

To evaluate SNL, we selected hls4ml [10] as the open-source counterpart in our comparative study. This mature HLS toolchain supports a broad range of network architectures, is actively maintained, and has been adopted in multiple scientific computing domains [11, 12]. Unlike SNL, hls4ml requires network weights and biases to be embedded into the FPGA fabric during synthesis. This limits runtime flexibility, but the framework compensates by offering user-configurable synthesis parameters, such as the strategy, enabling latency- or resource-optimized implementations, and the reuse factor, controlling the number of times a multiplier unit is reused during inference [13]. These trade-offs play a crucial role in applications where design constraints vary widely. As such, hls4ml serves as a well-established and representative comparison point, keeping our study informative yet tractable in size.

The comparison is structured as a benchmark measuring both inference latency and FPGA resource utilization across multiple network architectures and, when available, different hardware synthesis settings. In designing this benchmark, we account for multiple factors that critically impact performance, including data-flow patterns, synthesis strategies, and model complexity. These considerations, previously explored in earlier work [14], frame the trade-offs faced by real-time FPGA-based ML systems and motivate the extended analysis presented in this work. The benchmark spans both fully-connected neural networks (FCNNs) and convolutional neural networks (CNNs) from physics, audio, and vision domains. Results are reported in terms of logic, DSP, and BRAM consumption, as well as absolute latency ($\text{\,}\mathrm{\SIUnitSymbolMicro s}$), targeting a Zynq UltraScale+ MPSoC ZCU102 FPGA [15], with reports and designs generated using Vitis HLS and Vivado tools.

In what follows, we first provide background on both frameworks and their respective workflows on the ZCU102 board: section II describes SNL, Auto-SNL, and the integration with Rogue software on the MPSoC platform, whereas section III introduces hls4ml and explains its corresponding workflow on the ZCU102. These preliminary sections are followed by section IV, where we define the benchmarking protocol, including the models and the synthesis parameter design space. In section V and section VI, we present and analyze the comparative results between SNL and hls4ml across different hardware parameters, highlighting trade-offs in latency and resource utilization. Finally, section VII summarizes the key outcomes of this work, discussing limitations and directions for future work.

II SNL and Auto-SNL Workflow with SLAC’s Rogue Software

SNL is a high-level synthesis framework capable of deploying NNs into the programmable logic (PL) of FPGAs, with optimizations for ultra-low-latency inference in resource-constrained environments. SNL’s suitability for low-latency inference in demanding experiments has been demonstrated through its FPGA implementation of SpeckleNN [4], a deep learning model designed for real-time X-ray Single-Particle Imaging (SPI) at XFEL facilities. This application helps showcase SNL’s ability to execute scientific ML workloads close to the data source, reducing transfer and processing delays at the earliest stage of detector acquisition. Another key feature of SNL is its ability to dynamically update model weights and biases without FPGA re-synthesis, allowing models to be reconfigured and retrained on the fly. This is particularly advantageous for adaptive scientific experiments, where frequent updates and fine-tuning are required without interrupting real-time processing.

As shown through s 1 and 3, SNL deploys the NN layers entirely in PL for maximum efficiency. Weights and biases are loaded via AXI-Lite registers, while AXI-Stream facilitates real-time input and output data flow during inference. The direct memory access (DMA) engine ensures high-speed data transfer between processing elements, minimizing latency and optimizing performance.

The ZCU102 platform, running a custom PetaLinux image, manages device drivers, while SLAC’s Rogue software provides configurable hardware interaction [16]. This architecture, described in Figure 3, enables a streamlined, hardware-optimized pipeline for deploying neural networks on FPGAs, ensuring efficient real-time inference for scientific applications. A TCP stream bridge converts input and output data into TCP packets for Ethernet transmission, enabling seamless interaction with run control software.

To make the deployment of NN models on FPGA more accessible, we developed Auto-SNL, a complementary tool that simplifies the conversion of trained ML models into SNL-based HLS code. Auto-SNL bridges the gap between high-level frameworks like Keras and TensorFlow and SNL’s hardware-focused architecture. As illustrated in Figure 2, Auto-SNL requires a trained ML model and parameters such as data types, clock period, and the target FPGA board. Using these inputs, Auto-SNL writes the necessary files for building the SNL project on supported hardware. The generated project is then implemented to produce the bitfile, which can be integrated with the custom PetaLinux image and deployed on the FPGA. Throughout this process, Auto-SNL ensures compatibility with SNL and optimizes the generated code for the intended hardware platform.

One of the core objectives of Auto-SNL is to offer users the ability to adjust hardware parameters without requiring an in-depth understanding of the underlying HLS implementation. This allows fine-tuning of ML implementations, making it possible to tailor the deployment to specific requirements without manually modifying the generated code. Additionally, Auto-SNL aims to continue SNL’s modular design by providing a flexible and extensible workflow. Support for new hardware platforms and ML frameworks can be added by defining appropriate templates, ensuring that this tool adapts to evolving FPGA technologies and ML frameworks, and enabling rapid prototyping and deployment of ML inference systems.

III Workflow for deployment and inference on the ZCU102 using hls4ml

Figure 4 shows a standard hls4ml workflow for deploying NNs on the ZCU102 board configured with a PYNQ image [17]. Starting with a high-level model description (Keras, Pytorch, or ONNX) and a hardware-related configuration, hls4ml generates synthesizable C++ code, which is then packaged as a custom hardware IP core implementing the neural network in the PL.

During deployment, the custom IP connects to the Zynq processing system (PS) via AXI interfaces. An AXI-Lite interface provides control and status registers, while data movement can depend on hls4ml’s I/O type parameter. For IO Stream, the option matching SNL’s streaming interface, the IP core exchanges data with the PS through AXI-Stream, typically coupled with DMA engines to sustain high-throughput transfers between PS memory and the accelerator. Unlike SNL, where weights and biases can be updated at runtime, hls4ml embeds all model parameters into the PL during synthesis, meaning any retraining or parameter update requires regenerating and reloading a new bitstream.

Once the block design is implemented, the toolchain generates a bitstream file (.bit), defining the PL layout, including the NN accelerator, its interfacing logic, the embedded model parameters, and any other auxiliary IP. Alongside the bitstream, a hardware handoff file (.hwh) is produced, containing a machine-readable description of the block design, address map, and register definitions. On the PYNQ Linux OS running on the PS, these files are loaded at runtime through the Overlay API: the .bit configures the PL, while the .hwh enables automatic generation of Python-accessible drivers for controlling the accelerator and initiating data transfers.

At runtime, inference can be orchestrated from the PS, either through local Jupyter notebooks or remotely over Ethernet, with inputs and outputs exchanged using the data path defined at synthesis. This separation between hardware generation and runtime control allows hls4ml to combine FPGA-level acceleration with the flexibility of high-level Python interfaces, while giving developers control over a wide range of synthesis parameters, enabling fine-grained command over the latency-resource design space to suit diverse application requirements.

IV Benchmarking

To evaluate the performance and workflow characteristics of SNL, we conducted a comparative synthesis benchmark against the widely adopted hls4ml toolchain. The motivation for this study is twofold. First, the side-by-side evaluation enables a comprehensive assessment of latency and resource utilization across the two toolchains. Second, the results provide insight into where SNL’s current capabilities align with, or diverge from, established ML-to-FPGA workflows, thereby informing both user decision-making and potential directions for future development.

Our benchmark comprises four neural networks representing both domain-specific and general-purpose inference workloads. The selection includes (1) a particle jet classifier for high-energy physics applications [18], along with three models from the MLPerf Tiny benchmark suite [19]: (2) a fully connected autoencoder for anomaly detection in machine operating sounds, (3) a convolutional neural network for keyword spotting (KWS) in audio data, and (4) a convolution-based binary image classification model (VWW). This mix balances diversity in layer types, network depths, and parameter counts, ensuring coverage from compact to moderately complex architectures.

Table I summarizes each benchmark item, outlining the dataset used for training and validation, the input size, the intended task, and the performance achieved, measured with either accuracy or area-under-curve (AUC) metrics. All networks were trained on their respective datasets following the pre-processing and input formatting procedures described in the original references [18, 19]. To ensure compatibility with the ZCU102 board and SNL’s current feature set, architectures and input sizes, particularly for the convolutional networks, were adjusted to reduce resource usage and align with SNL’s layer support. For classification networks, we also truncate the final softmax activation layer during hardware synthesis. Table III illustrates the full network architectures as well as the training hyperparameters.

In terms of hardware synthesis, our parameter space attempts to capture common design trade-offs in FPGA-based ML acceleration while keeping the experiment tractable. In this context, the first synthesis parameter we considered is precision, referring to the bit-width used to represent the inputs and layer weights, which directly impacts resource usage and on-chip numerical accuracy. For both SNL and hls4ml, three representative fixed-point precisions were selected to span high-accuracy, balanced, and resource-efficient operating points. We also varied two additional parameters specific to hls4ml, strategy and reuse factor. The strategy setting determines whether synthesis prioritizes minimizing latency or conserving FPGA resources. The reuse factor (RF) controls the degree of operator reuse: up to a certain point, higher RF values result in lower parallelism and lower resource use, but increased latency. The complete set of synthesis parameters and their values is summarized in Table II.

Precision values are denoted as $\langle X,Y\rangle$, where $X$ is the total bit-width and $Y$ is the number of bits representing the signed number above the binary point. In our experiment, we omit RF = 1 in the resource-optimized strategy, following hls4ml’s recommendation to avoid this specific combination [24].

All designs targeted the Xilinx Zynq UltraScale+ MPSoC ZCU102 board using Vitis HLS and Vivado 2023.1 [25]. For an unbiased and consistent comparison, we set the hls4ml IO type parameter to Stream, matching SNL’s streaming interface. For SNL, the reported latency excludes the one-time streaming transfer of model weights, which are loaded before inference begins. Therefore, for both frameworks, latency is defined as the time between the arrival of the first input at the FPGA’s streaming interface and the production of the corresponding output. Latency results are obtained from the C-synthesis timing analysis, while resource utilization values (BRAM, DSP, FF, LUT) are taken from the post-implementation report.

V Results

We benchmarked multiple neural network architectures using both SNL and hls4ml, running Vitis HLS and Vivado synthesis with the ZCU102 board as the target. The following shows the benchmark results, with synthesis parameters and configurations as described in section IV.

s 5 and 6 present the synthesis results grouped by neural network model and fixed-point precision. Colors indicate the model architecture. For each model–precision pair, the leftmost shaded bar corresponds to the SNL implementation, while the subsequent non-shaded bars show hls4ml implementations ordered by synthesis strategy—latency (L) first, then resource (R)—and by increasing RF. In total, 12 groups are expected (4 architectures, 3 precisions), and each group should contain 1 SNL synthesis followed by 7 hls4ml variants (L/R $\times$ RF).

The vertical axis in Figure 5 reports absolute FPGA resource usage as given by the implementation report: LUT, FF, DSP, and BRAM counts, expressed in absolute numbers. The vertical axis in Figure 6 reports absolute inference latency in microseconds, as reported by C-synthesis. All latency values are device- and clock-specific.

Missing groups in the bar plots indicate synthesis failure of a given model-precision combination with SNL, mainly due to resource limitations. Meanwhile, absent hls4ml bars within the same group reflect hls4ml designs that failed synthesis for the same reason. This layout allows quick comparison between the single SNL design and multiple hls4ml configurations, while making synthesis success and failure immediately apparent.

VI Discussion

The synthesis results highlight clear trade-offs between SNL and hls4ml, reflecting both the frameworks’ differing implementation choices and the latency-resource patterns observed across the model-parameter space.

In terms of resource utilization, a high-level view of Figure 5 shows that SNL consistently uses more BRAM and mostly higher FF counts than hls4ml across models and parameters. Meanwhile, DSP and LUT consumption exhibit an architecture-dependent behavior: for CNN models, SNL typically requires more logic resources than hls4ml, whereas for FCNNs, SNL employs fewer LUTs across all precisions and fewer DSPs in higher ones. Conversely, hls4ml demonstrates a consistent and significant decrease in DSP utilization in lower precisions, particularly with precision $\langle 8,3\rangle$. As expected, increasing hls4ml’s reuse factor, which determines the number of times the same multiplier unit is reused, consistently lowers DSP usage. LUTs also show a similar trend within the resource-focused strategy.

As noted in section V, the absence of certain groups in Figure 5 reflects SNL synthesis failures for specific model-precision combinations. For instance, KWS-$\langle 32,16\rangle$ synthesis reports a failure due to exceeding the available ZCU102 DSP and LUT units, while VWW-$\langle 8,3\rangle$ failure stems from a framework-related issue.

Similarly, missing hls4ml bars within an otherwise present group correspond to configurations that failed. In our benchmark set, hls4ml fails to implement the Anomaly autoencoder for precisions $\langle 32,16\rangle$, and $\langle 16,6\rangle$, as well as the fully-connected Jet model with precision $\langle 32,16\rangle$ and RFs 1 and 2. Post-synthesis reports attribute these failures to exceeding either DSP or LUT availability.

In terms of latency, the trends in Figure 6, presented in logarithmic scale, show that SNL achieves lower inference latency than hls4ml in three out of the four benchmarked architectures, with the gap widening as RF increases, as expected. This effect is most pronounced for the CNNs, where the higher RF values in hls4ml, especially under the resource-focused strategy, cause substantial latency penalties due to multiplier reuse in convolution loops. The exception is the Jet architecture, where lower complexity and reduced per-layer parameter count potentially allow hls4ml to maintain a latency lower than SNL, with minimal variation across parameters.

Overall, the results, while constrained to our benchmark’s limited set of models and synthesis configurations, demonstrate that SNL offers promising latency advantages for most tested architectures, often at the cost of higher BRAM and FF usage. For the FCNNs in particular, SNL also achieves notable DSP and LUT savings compared to hls4ml, especially at higher precisions. By contrast, hls4ml currently provides finer-grained control over resource–latency trade-offs through adjustable parameters beyond precision.

VII Conclusion

Deploying neural networks in high-rate experimental environments demands low latency, efficient resource usage, and streamlined design flows. SNL is designed to meet these challenges by providing a specialized ML-to-FPGA framework with a workflow tailored for fast iteration. Through dynamic weight and bias reloading, SNL reduces the need for repeated synthesis when models change, enabling rapid adaptation to evolving experimental conditions.

Auto-SNL is a key addition we presented in this work that bridges Python-based model definition and hardware synthesis by automatically translating network architectures into HLS code and running the necessary implementation steps. This allows domain experts to deploy and refine models without delving into low-level C++ design, greatly shortening development cycles.

To evaluate the framework along with the full workflow, we performed a benchmark comparison against hls4ml across multiple neural network architectures, fixed-point precisions, and synthesis configurations targeting a Xilinx ZCU102 FPGA. The results showed that SNL achieves competitive or superior latency in most tested architectures, while in some cases also offering FPGA resource savings. At the same time, it is clear that SNL currently lacks an extensive parameter space outside of input and weight precision, limiting user control over the latency and resource utilization trade-offs. Additionally, the observed trends, while clear, are restricted to our non-exhaustive benchmark set and might not generalize to other models and deployment targets.

Looking forward, we plan to investigate a broader range of models and synthesis parameters, and target additional FPGA boards, expanding our current benchmark and further confirming the advantages SNL showcased in this work. Future work will also benefit from running the synthesized models on actual hardware, measuring on-chip inference latency and throughput to complement the results from post-synthesis reports. Finally, parallel to SNL’s development, we aim to update Auto-SNL, exposing additional synthesis parameters and potentially offering users a graphical interface to streamline synthesis configuration. These developments will position SNL as a versatile yet accessible framework for real-time embedded neural network deployment, particularly in high-rate, resource-constrained environments.

Acknowledgment

Abhilasha Dave, Larry Ruckman, James J. Russell, and Ryan Herbst’s work was supported by the U.S. Department of Energy, under contract number DE-AC02-76SF0051. We would also like to extend our thanks to CMC Microsystems for access to Xilinx software licenses, as well as 3IT’s GRAMS group for generously providing the hardware that helped make this work possible.