Horizontal and Vertical Attention in Transformers

TL;DR

This paper introduces horizontal and vertical attention mechanisms to enhance feature representation in Transformers, improving their performance and generalization with minimal additional computational cost.

Contribution

It proposes modular horizontal and vertical attention modules that can be integrated into Transformers to improve feature re-weighting and channel-wise calibration.

Findings

Enhanced Transformer models show improved performance across tasks.

The proposed attentions require minimal additional computation.

The modules are highly modular and adaptable.

Abstract

Transformers are built upon multi-head scaled dot-product attention and positional encoding, which aim to learn the feature representations and token dependencies. In this work, we focus on enhancing the distinctive representation by learning to augment the feature maps with the self-attention mechanism in Transformers. Specifically, we propose the horizontal attention to re-weight the multi-head output of the scaled dot-product attention before dimensionality reduction, and propose the vertical attention to adaptively re-calibrate channel-wise feature responses by explicitly modelling inter-dependencies among different channels. We demonstrate the Transformer models equipped with the two attentions have a high generalization capability across different supervised learning tasks, with a very minor additional computational cost overhead. The proposed horizontal and vertical attentions are highly modular, which can be inserted into various Transformer models to further improve the performance. Our code is available in the supplementary material.