Fair Comparison between Efficient Attentions

TL;DR

This paper systematically compares various efficient attention mechanisms in transformers for image classification, isolating their performance differences by standardizing model configurations and training schemes.

Contribution

It provides a direct, fair comparison of efficient attention methods on ImageNet1K, highlighting which approaches perform best under identical conditions.

Findings

Certain efficient attention methods outperform others in classification accuracy.

Trade-offs between computational efficiency and accuracy are identified.

Standardized evaluation framework for future attention model comparisons.

Abstract

Transformers have been successfully used in various fields and are becoming the standard tools in computer vision. However, self-attention, a core component of transformers, has a quadratic complexity problem, which limits the use of transformers in various vision tasks that require dense prediction. Many studies aiming at solving this problem have been reported proposed. However, no comparative study of these methods using the same scale has been reported due to different model configurations, training schemes, and new methods. In our paper, we validate these efficient attention models on the ImageNet1K classification task by changing only the attention operation and examining which efficient attention is better.