Attention as Activation

TL;DR

This paper introduces attentional activation (ATAC) units that unify activation functions and attention mechanisms, leading to improved neural network performance with modest additional parameters.

Contribution

The paper proposes a novel attentional activation unit that combines activation and attention, and demonstrates its effectiveness across multiple datasets and network architectures.

Findings

ATAC units improve network performance over traditional activations.

Networks with ATAC units outperform competitors with similar parameter counts.

Empirical validation on CIFAR-10, CIFAR-100, and ImageNet datasets confirms effectiveness.

Abstract

Activation functions and attention mechanisms are typically treated as having different purposes and have evolved differently. However, both concepts can be formulated as a non-linear gating function. Inspired by their similarity, we propose a novel type of activation units called attentional activation (ATAC) units as a unification of activation functions and attention mechanisms. In particular, we propose a local channel attention module for the simultaneous non-linear activation and element-wise feature refinement, which locally aggregates point-wise cross-channel feature contexts. By replacing the well-known rectified linear units by such ATAC units in convolutional networks, we can construct fully attentional networks that perform significantly better with a modest number of additional parameters. We conducted detailed ablation studies on the ATAC units using several host networks with varying network depths to empirically verify the effectiveness and efficiency of the units. Furthermore, we compared the performance of the ATAC units against existing activation functions as well as other attention mechanisms on the CIFAR-10, CIFAR-100, and ImageNet datasets. Our experimental results show that networks constructed with the proposed ATAC units generally yield performance gains over their competitors given a comparable number of parameters.