Low Latency Transformer Inference on FPGAs for Physics Applications with hls4ml

TL;DR

This paper introduces an FPGA-based implementation of transformer models using hls4ml, achieving sub-2 microsecond latency suitable for real-time physics applications, and demonstrating scalability across models.

Contribution

It presents a novel FPGA implementation of transformers with optimized layers, enabling real-time inference with high scalability using hls4ml.

Findings

Latency less than 2 microseconds on FPGA

Compatibility with TensorFlow transformer models

Potential for real-time physics applications

Abstract

This study presents an efficient implementation of transformer architectures in Field-Programmable Gate Arrays(FPGAs) using hls4ml. We demonstrate the strategy for implementing the multi-head attention, softmax, and normalization layer and evaluate three distinct models. Their deployment on VU13P FPGA chip achieved latency less than 2us, demonstrating the potential for real-time applications. HLS4ML compatibility with any TensorFlow-built transformer model further enhances the scalability and applicability of this work. Index Terms: FPGAs, machine learning, transformers, high energy physics, LIGO