BitStopper: An Efficient Transformer Attention Accelerator via Stage-fusion and Early Termination

TL;DR

BitStopper introduces a novel transformer attention acceleration method that combines stage-fusion and early termination techniques, significantly reducing computation and memory costs in large language models.

Contribution

It presents a new hardware-efficient algorithm-architecture co-design that eliminates the need for a sparsity predictor and enhances performance through stage fusion and adaptive token selection.

Findings

Achieves over 2x speedup compared to state-of-the-art accelerators.

Delivers more than 2x improvements in energy efficiency.

Effectively reduces memory access and computation in transformer models.

Abstract

Attention-based large language models (LLMs) have transformed modern AI applications, but the quadratic cost of self-attention imposes significant compute and memory overhead. Dynamic sparsity (DS) attention mitigates this, yet its hardware efficiency is limited by the added prediction stage and the heavy memory traffic it entails. To address these limitations, this paper proposes BitStopper, a fine-grained algorithm-architecture co-design that operates without a sparsity predictor. First, a bit-serial enable stage fusion (BESF) mechanism is proposed to reuse and minimize the memory access by progressively terminating trivial tokens and merging the prediction stage into the execution stage. Second, a lightweight and adaptive token selection (LATS) strategy is developed to work in concert with the bit-level sparsity speculation. Third, a bit-level asynchronous processing (BAP) strategy is employed to improve compute utilization during the on-demand bit-grained memory fetching. Finally, an elaborate architecture is designed to translate the theoretical complexity reduction into practical performance improvement. Extensive evaluations demonstrate that, compared to state-of-the-art (SOTA) Transformer accelerators, BitStopper achieves 2.03x and 1.89x speedups over Sanger and SOFA, respectively, while delivering 2.4x and 2.1x improvements in energy efficiency.