Generalized Attention Mechanism and Relative Position for Transformer

TL;DR

This paper introduces a generalized attention mechanism (GAM) for transformers, offering a new interpretation, variants, and a flexible relative position representation suitable for sequences with non-adjacent elements.

Contribution

It presents a novel generalized attention framework with a new relative position encoding, enhancing transformer flexibility and applicability to diverse sequence data.

Findings

Proposes a new interpretation of self-attention.

Develops various attention variants within GAM.

Introduces a flexible relative position representation.

Abstract

In this paper, we propose generalized attention mechanism (GAM) by first suggesting a new interpretation for self-attention mechanism of Vaswani et al. . Following the interpretation, we provide description for different variants of attention mechanism which together form GAM. Further, we propose a new relative position representation within the framework of GAM. This representation can be easily utilized for cases in which elements next to each other in input sequence can be at random locations in actual dataset/corpus.