EAANet: Efficient Attention Augmented Convolutional Networks

TL;DR

EAANet introduces an efficient self-attention mechanism within a hybrid convolutional architecture, significantly reducing memory usage and improving high-resolution image processing compared to traditional self-attention models.

Contribution

The paper proposes EAANet, a novel hybrid architecture that incorporates efficient self-attention to enhance performance and scalability in convolutional networks.

Findings

EAANet outperforms AA-Net and ResNet18 in accuracy.

Efficient self-attention scales better with input size.

Training hybrid models is more challenging than standard ResNet.

Abstract

Humans can effectively find salient regions in complex scenes. Self-attention mechanisms were introduced into Computer Vision (CV) to achieve this. Attention Augmented Convolutional Network (AANet) is a mixture of convolution and self-attention, which increases the accuracy of a typical ResNet. However, The complexity of self-attention is O(n2) in terms of computation and memory usage with respect to the number of input tokens. In this project, we propose EAANet: Efficient Attention Augmented Convolutional Networks, which incorporates efficient self-attention mechanisms in a convolution and self-attention hybrid architecture to reduce the model's memory footprint. Our best model show performance improvement over AA-Net and ResNet18. We also explore different methods to augment Convolutional Network with self-attention mechanisms and show the difficulty of training those methods compared to ResNet. Finally, we show that augmenting efficient self-attention mechanisms with ResNet scales better with input size than normal self-attention mechanisms. Therefore, our EAANet is more capable of working with high-resolution images.