Introduce the Result Into Self-Attention

TL;DR

This paper introduces a novel self-attention modification that incorporates precomputed classification results into the attention mechanism, leading to improved accuracy in image classification tasks.

Contribution

It proposes using auxiliary classifier outputs as input to attention networks, enhancing traditional self-attention methods in convolutional neural networks.

Findings

Achieved up to 1.94% accuracy improvement on CIFAR-100

Demonstrated effectiveness of auxiliary classifier integration in attention mechanisms

Enhanced performance of SE-ResNet with the proposed method

Abstract

Traditional self-attention mechanisms in convolutional networks tend to use only the output of the previous layer as input to the attention network, such as SENet, CBAM, etc. In this paper, we propose a new attention modification method that tries to get the output of the classification network in advance and use it as a part of the input of the attention network. We used the auxiliary classifier proposed in GoogLeNet to obtain the results in advance and pass them into attention networks. we added this mechanism to SE-ResNet for our experiments and achieved a classification accuracy improvement of at most 1.94% on cifar100.