ProTEA: Programmable Transformer Encoder Acceleration on FPGA

TL;DR

ProTEA is a programmable FPGA-based accelerator designed for transformer encoder models, achieving significant speedups over GPUs and existing FPGA solutions by optimizing parallelism and matrix tiling.

Contribution

This paper presents ProTEA, a flexible, runtime-programmable FPGA accelerator specifically optimized for dense transformer encoder computations, with novel tiling strategies for improved performance.

Findings

ProTEA achieves 2.5× faster inference than NVIDIA Titan XP.

ProTEA outperforms current FPGA accelerators by 1.3–2.8×.

ProTEA supports a wide range of transformer models with near-optimal performance.

Abstract

Transformer neural networks (TNN) have been widely utilized on a diverse range of applications, including natural language processing (NLP), machine translation, and computer vision (CV). Their widespread adoption has been primarily driven by the exceptional performance of their multi-head self-attention block used to extract key features from sequential data. The multi-head self-attention block is followed by feedforward neural networks, which play a crucial role in introducing non-linearity to assist the model in learning complex patterns. Despite the popularity of TNNs, there has been limited numbers of hardware accelerators targeting these two critical blocks. Most prior works have concentrated on sparse architectures that are not flexible for popular TNN variants. This paper introduces \textit{ProTEA}, a runtime programmable accelerator tailored for the dense computations of most of state-of-the-art transformer encoders. \textit{ProTEA} is designed to reduce latency by maximizing parallelism. We introduce an efficient tiling of large matrices that can distribute memory and computing resources across different hardware components within the FPGA. We provide run time evaluations of \textit{ProTEA} on a Xilinx Alveo U55C high-performance data center accelerator card. Experimental results demonstrate that \textit{ProTEA} can host a wide range of popular transformer networks and achieve near optimal performance with a tile size of 64 in the multi-head self-attention block and 6 in the feedforward networks block when configured with 8 parallel attention heads, 12 layers, and an embedding dimension of 768 on the U55C. Comparative results are provided showing \textit{ProTEA} is 2.5$\times$ faster than an NVIDIA Titan XP GPU. Results also show that it achieves 1.3 -- 2.8$\times$ speed up compared with current state-of-the-art custom designed FPGA accelerators.