Flash Window Attention: speedup the attention computation for Swin Transformer

TL;DR

This paper introduces Flash Window Attention, an optimized attention mechanism for Swin Transformer that significantly speeds up computation and improves runtime efficiency by tailoring flash attention for window-based processing.

Contribution

The paper presents a novel Flash Window Attention method that adapts flash attention for windowed image processing, achieving up to 300% speedup and 30% runtime improvement.

Findings

Attention computation speed increased by up to 300%.

End-to-end runtime efficiency improved by up to 30%.

Code implementation is publicly available.

Abstract

To address the high resolution of image pixels, the Swin Transformer introduces window attention. This mechanism divides an image into non-overlapping windows and restricts attention computation to within each window, significantly enhancing computational efficiency. To further optimize this process, one might consider replacing standard attention with flash attention, which has proven to be more efficient in language models. However, a direct substitution is ineffective. Flash attention is designed for long sequences, whereas window attention deals with shorter sequences but must handle numerous of them in parallel. In this report, we present an optimized solution called Flash Window Attention, tailored specifically for window attention. Flash Window Attention improves attention computation efficiency by up to 300% and enhances end-to-end runtime efficiency by up to 30%. Our code is available online.