SHAPE: Shifted Absolute Position Embedding for Transformers

TL;DR

SHAPE introduces a simple and efficient shifted absolute position embedding method for Transformers that enhances shift invariance and generalization to unseen sequence lengths.

Contribution

The paper proposes SHAPE, a novel position embedding technique that improves shift invariance and computational efficiency in Transformers.

Findings

SHAPE achieves comparable performance to existing methods.

SHAPE is simpler and faster to compute.

SHAPE enhances generalization to unseen sequence lengths.

Abstract

Position representation is crucial for building position-aware representations in Transformers. Existing position representations suffer from a lack of generalization to test data with unseen lengths or high computational cost. We investigate shifted absolute position embedding (SHAPE) to address both issues. The basic idea of SHAPE is to achieve shift invariance, which is a key property of recent successful position representations, by randomly shifting absolute positions during training. We demonstrate that SHAPE is empirically comparable to its counterpart while being simpler and faster.