A Runtime-Adaptive Transformer Neural Network Accelerator on FPGAs

TL;DR

This paper presents ADAPTOR, a runtime-adaptive FPGA accelerator for transformer neural networks that improves efficiency and speed across various models and platforms, addressing the high computational demands of TNNs.

Contribution

The paper introduces ADAPTOR, a novel FPGA-based accelerator that adapts at runtime for dense matrix computations in transformer models, enhancing efficiency and flexibility.

Findings

Achieves 1.2× power efficiency over NVIDIA K80 GPU.

Achieves 2.87× power efficiency over i7-8700K CPU.

Provides 1.7 to 2.25× speedup over existing FPGA accelerators.

Abstract

Transformer neural networks (TNN) excel in natural language processing (NLP), machine translation, and computer vision (CV) without relying on recurrent or convolutional layers. However, they have high computational and memory demands, particularly on resource-constrained devices like FPGAs. Moreover, transformer models vary in processing time across applications, requiring custom models with specific parameters. Designing custom accelerators for each model is complex and time-intensive. Some custom accelerators exist with no runtime adaptability, and they often rely on sparse matrices to reduce latency. However, hardware designs become more challenging due to the need for application-specific sparsity patterns. This paper introduces ADAPTOR, a runtime-adaptive accelerator for dense matrix computations in transformer encoders and decoders on FPGAs. ADAPTOR enhances the utilization of processing elements and on-chip memory, enhancing parallelism and reducing latency. It incorporates efficient matrix tiling to distribute resources across FPGA platforms and is fully quantized for computational efficiency and portability. Evaluations on Xilinx Alveo U55C data center cards and embedded platforms like VC707 and ZCU102 show that our design is 1.2$\times$ and 2.87$\times$ more power efficient than the NVIDIA K80 GPU and the i7-8700K CPU respectively. Additionally, it achieves a speedup of 1.7 to 2.25$\times$ compared to some state-of-the-art FPGA-based accelerators.