HDAM: Heuristic Difference Attention Module for Convolutional Neural Networks

TL;DR

This paper introduces HDAM, a novel attention mechanism for CNNs that recalibrates inputs based on the difference between local and global contextual information, optimized with genetic algorithms for receptive field size.

Contribution

The paper proposes HDAM, a new attention module based on difference of contextual info, and uses genetic algorithms to optimize receptive field sizes in CNNs.

Findings

HDAM achieves higher accuracy with fewer parameters on CIFAR datasets.

Genetic algorithm effectively optimizes local receptive field sizes.

HDAM outperforms traditional attention mechanisms in experiments.

Abstract

The attention mechanism is one of the most important priori knowledge to enhance convolutional neural networks. Most attention mechanisms are bound to the convolutional layer and use local or global contextual information to recalibrate the input. This is a popular attention strategy design method. Global contextual information helps the network to consider the overall distribution, while local contextual information is more general. The contextual information makes the network pay attention to the mean or maximum value of a particular receptive field. Different from the most attention mechanism, this article proposes a novel attention mechanism with the heuristic difference attention module, HDAM. HDAM's input recalibration is based on the difference between the local and global contextual information instead of the mean and maximum values. At the same time, to make different layers have a more suitable local receptive field size and increase the exibility of the local receptive field design, we use genetic algorithm to heuristically produce local receptive fields. First, HDAM extracts the mean value of the global and local receptive fields as the corresponding contextual information. Then the difference between the global and local contextual information is calculated. Finally HDAM uses this difference to recalibrate the input. In addition, we use the heuristic ability of genetic algorithm to search for the local receptive field size of each layer. Our experiments on CIFAR-10 and CIFAR-100 show that HDAM can use fewer parameters than other attention mechanisms to achieve higher accuracy. We implement HDAM with the Python library, Pytorch, and the code and models will be publicly available.