Swin-Free: Achieving Better Cross-Window Attention and Efficiency with Size-varying Window

TL;DR

Swin-Free introduces size-varying windows across stages in transformer models to improve cross-window connectivity and efficiency, outperforming Swin Transformer in speed and accuracy without shifting windows.

Contribution

The paper proposes a novel size-varying window approach that replaces shifting windows in Swin Transformer, enhancing efficiency and accuracy in vision tasks.

Findings

Swin-Free runs faster than Swin Transformer at inference.

Swin-Free achieves better accuracy than Swin Transformer.

Variants of Swin-Free are also faster than their Swin counterparts.

Abstract

Transformer models have shown great potential in computer vision, following their success in language tasks. Swin Transformer is one of them that outperforms convolution-based architectures in terms of accuracy, while improving efficiency when compared to Vision Transformer (ViT) and its variants, which have quadratic complexity with respect to the input size. Swin Transformer features shifting windows that allows cross-window connection while limiting self-attention computation to non-overlapping local windows. However, shifting windows introduces memory copy operations, which account for a significant portion of its runtime. To mitigate this issue, we propose Swin-Free in which we apply size-varying windows across stages, instead of shifting windows, to achieve cross-connection among local windows. With this simple design change, Swin-Free runs faster than the Swin Transformer at inference with better accuracy. Furthermore, we also propose a few of Swin-Free variants that are faster than their Swin Transformer counterparts.