Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERT

TL;DR

This paper introduces FCA, a hybrid self-attention mechanism for BERT that reduces computational cost by dynamically switching between fine- and coarse-grained tokens, maintaining accuracy while halving FLOPs.

Contribution

FCA is a novel hybrid self-attention method that adaptively shortens sequence length, improving efficiency without significant accuracy loss in BERT models.

Findings

FCA achieves 2x reduction in FLOPs on benchmarks.

FCA maintains less than 1% accuracy loss.

Outperforms prior methods in accuracy-FLOPs trade-off.

Abstract

Transformer-based pre-trained models, such as BERT, have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications. However, deploying these models can be prohibitively costly, as the standard self-attention mechanism of the Transformer suffers from quadratic computational cost in the input sequence length. To confront this, we propose FCA, a fine- and coarse-granularity hybrid self-attention that reduces the computation cost through progressively shortening the computational sequence length in self-attention. Specifically, FCA conducts an attention-based scoring strategy to determine the informativeness of tokens at each layer. Then, the informative tokens serve as the fine-granularity computing units in self-attention and the uninformative tokens are replaced with one or several clusters as the coarse-granularity computing units in self-attention. Experiments on GLUE and RACE datasets show that BERT with FCA achieves 2x reduction in FLOPs over original BERT with <1% loss in accuracy. We show that FCA offers a significantly better trade-off between accuracy and FLOPs compared to prior methods.