SWAT: Scalable and Efficient Window Attention-based Transformers Acceleration on FPGAs

TL;DR

This paper introduces SWAT, an FPGA-based accelerator that efficiently handles long-context Transformer models using window attention, achieving significant improvements in latency and energy efficiency.

Contribution

The paper presents a novel dataflow-aware FPGA microarchitecture optimized for sparse window attention, enabling scalable and efficient long-input processing.

Findings

Up to 22× latency reduction compared to baseline FPGA accelerators.

Up to 5.7× energy efficiency improvement over baseline FPGA.

15× energy efficiency gain over GPU-based solutions.

Abstract

Efficiently supporting long context length is crucial for Transformer models. The quadratic complexity of the self-attention computation plagues traditional Transformers. Sliding window-based static sparse attention mitigates the problem by limiting the attention scope of the input tokens, reducing the theoretical complexity from quadratic to linear. Although the sparsity induced by window attention is highly structured, it does not align perfectly with the microarchitecture of the conventional accelerators, leading to suboptimal implementation. In response, we propose a dataflow-aware FPGA-based accelerator design, SWAT, that efficiently leverages the sparsity to achieve scalable performance for long input. The proposed microarchitecture is based on a design that maximizes data reuse by using a combination of row-wise dataflow, kernel fusion optimization, and an input-stationary design considering the distributed memory and computation resources of FPGA. Consequently, it achieves up to 22$\times$ and 5.7$\times$ improvement in latency and energy efficiency compared to the baseline FPGA-based accelerator and 15$\times$ energy efficiency compared to GPU-based solution.