Ultra Fast Transformers on FPGAs for Particle Physics Experiments

TL;DR

This paper presents an FPGA implementation of transformer components optimized for particle physics experiments, achieving ultra-low latency suitable for real-time trigger systems at CERN.

Contribution

It introduces an efficient FPGA-based transformer implementation using hls4ml, tailored for particle physics applications, with critical components like multi-head attention and softmax.

Findings

Latency under 2 microseconds on Xilinx UltraScale+ FPGA

Effective application to jet flavor tagging problem

Compatible with CERN LHC trigger requirements

Abstract

This work introduces a highly efficient implementation of the transformer architecture on a Field-Programmable Gate Array (FPGA) by using the \texttt{hls4ml} tool. Given the demonstrated effectiveness of transformer models in addressing a wide range of problems, their application in experimental triggers within particle physics becomes a subject of significant interest. In this work, we have implemented critical components of a transformer model, such as multi-head attention and softmax layers. To evaluate the effectiveness of our implementation, we have focused on a particle physics jet flavor tagging problem, employing a public dataset. We recorded latency under 2 $\mu$s on the Xilinx UltraScale+ FPGA, which is compatible with hardware trigger requirements at the CERN Large Hadron Collider experiments.