SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequences

TL;DR

SALO is a specialized hardware accelerator that efficiently implements hybrid sparse attention mechanisms, significantly reducing computation time for long sequence processing in transformer models.

Contribution

It introduces a novel spatial accelerator with a data scheduler for hybrid sparse attention, addressing performance issues in long sequence tasks.

Findings

Achieves 17.66x speedup over GPU

Achieves 89.33x speedup over CPU

Effective for Longformer and ViL workloads

Abstract

The attention mechanisms of transformers effectively extract pertinent information from the input sequence. However, the quadratic complexity of self-attention w.r.t the sequence length incurs heavy computational and memory burdens, especially for tasks with long sequences. Existing accelerators face performance degradation in these tasks. To this end, we propose SALO to enable hybrid sparse attention mechanisms for long sequences. SALO contains a data scheduler to map hybrid sparse attention patterns onto hardware and a spatial accelerator to perform the efficient attention computation. We show that SALO achieves 17.66x and 89.33x speedup on average compared to GPU and CPU implementations, respectively, on typical workloads, i.e., Longformer and ViL.