Self-Supervised Implicit Attention: Guided Attention by The Model Itself

TL;DR

This paper introduces Self-Supervised Implicit Attention (SSIA), a training-only attention mechanism that enhances neural network performance without extra inference costs by leveraging hierarchical features for self-guided supervision.

Contribution

The paper presents SSIA, a novel self-supervised attention method that does not add inference overhead and improves model accuracy by using hierarchical features for training guidance.

Findings

SSIA significantly improves classification accuracy.

Outperforms popular attention methods like SE and CBAM.

No additional inference costs incurred.

Abstract

We propose Self-Supervised Implicit Attention (SSIA), a new approach that adaptively guides deep neural network models to gain attention by exploiting the properties of the models themselves. SSIA is a novel attention mechanism that does not require any extra parameters, computation, or memory access costs during inference, which is in contrast to existing attention mechanism. In short, by considering attention weights as higher-level semantic information, we reconsidered the implementation of existing attention mechanisms and further propose generating supervisory signals from higher network layers to guide lower network layers for parameter updates. We achieved this by building a self-supervised learning task using the hierarchical features of the network itself, which only works at the training stage. To verify the effectiveness of SSIA, we performed a particular implementation (called an SSIA block) in convolutional neural network models and validated it on several image classification datasets. The experimental results show that an SSIA block can significantly improve the model performance, even outperforms many popular attention methods that require additional parameters and computation costs, such as Squeeze-and-Excitation and Convolutional Block Attention Module. Our implementation will be available on GitHub.