GAttANet: Global attention agreement for convolutional neural networks

TL;DR

GAttANet introduces a brain-inspired global attention mechanism that enhances convolutional neural networks by pooling spatial key-query vectors into a global attention query, modulating activations based on agreement, and improving accuracy across various architectures and datasets.

Contribution

This paper presents a novel global attention system inspired by biological brains, integrated into CNNs to improve performance with minimal additional parameters.

Findings

Improves accuracy on CIFAR10, CIFAR100, and ImageNet-1k datasets.

Effective across different CNN architectures, including ResNet50.

Requires relatively few additional parameters.

Abstract

Transformer attention architectures, similar to those developed for natural language processing, have recently proved efficient also in vision, either in conjunction with or as a replacement for convolutional layers. Typically, visual attention is inserted in the network architecture as a (series of) feedforward self-attention module(s), with mutual key-query agreement as the main selection and routing operation. However efficient, this strategy is only vaguely compatible with the way that attention is implemented in biological brains: as a separate and unified network of attentional selection regions, receiving inputs from and exerting modulatory influence on the entire hierarchy of visual regions. Here, we report experiments with a simple such attention system that can improve the performance of standard convolutional networks, with relatively few additional parameters. Each spatial position in each layer of the network produces a key-query vector pair; all queries are then pooled into a global attention query. On the next iteration, the match between each key and the global attention query modulates the network's activations -- emphasizing or silencing the locations that agree or disagree (respectively) with the global attention system. We demonstrate the usefulness of this brain-inspired Global Attention Agreement network (GAttANet) for various convolutional backbones (from a simple 5-layer toy model to a standard ResNet50 architecture) and datasets (CIFAR10, CIFAR100, Imagenet-1k). Each time, our global attention system improves accuracy over the corresponding baseline.