MaiT: Leverage Attention Masks for More Efficient Image Transformers

TL;DR

MaiT introduces attention masks to image transformers, effectively incorporating spatial locality to improve accuracy and efficiency, making them more suitable for embedded applications.

Contribution

The paper proposes a novel attention masking technique that encodes locality in image transformers, enhancing performance and efficiency across various architectures.

Findings

Top-1 accuracy increases by up to 1.7% over CaiT.

Model parameters and FLOPs are reduced.

Throughput improves by up to 1.5X compared to Swin.

Abstract

Though image transformers have shown competitive results with convolutional neural networks in computer vision tasks, lacking inductive biases such as locality still poses problems in terms of model efficiency especially for embedded applications. In this work, we address this issue by introducing attention masks to incorporate spatial locality into self-attention heads. Local dependencies are captured efficiently with masked attention heads along with global dependencies captured by unmasked attention heads. With Masked attention image Transformer - MaiT, top-1 accuracy increases by up to 1.7% compared to CaiT with fewer parameters and FLOPs, and the throughput improves by up to 1.5X compared to Swin. Encoding locality with attention masks is model agnostic, and thus it applies to monolithic, hierarchical, or other novel transformer architectures.