Global Attention Mechanism: Retain Information to Enhance Channel-Spatial Interactions

TL;DR

This paper introduces a global attention mechanism that preserves information across channel and spatial dimensions, improving deep neural network performance in image classification tasks by enhancing cross-dimension interactions.

Contribution

It proposes a novel global attention mechanism combining 3D-permutation with MLP for channel attention and a convolutional spatial attention module, addressing information loss in prior methods.

Findings

Outperforms recent attention mechanisms on CIFAR-100 and ImageNet-1K

Improves performance of ResNet and MobileNet architectures

Demonstrates stable and consistent performance gains

Abstract

A variety of attention mechanisms have been studied to improve the performance of various computer vision tasks. However, the prior methods overlooked the significance of retaining the information on both channel and spatial aspects to enhance the cross-dimension interactions. Therefore, we propose a global attention mechanism that boosts the performance of deep neural networks by reducing information reduction and magnifying the global interactive representations. We introduce 3D-permutation with multilayer-perceptron for channel attention alongside a convolutional spatial attention submodule. The evaluation of the proposed mechanism for the image classification task on CIFAR-100 and ImageNet-1K indicates that our method stably outperforms several recent attention mechanisms with both ResNet and lightweight MobileNet.