Alternative positional encoding functions for neural transformers

TL;DR

This paper proposes alternative periodic functions for positional encoding in neural transformers, showing they can outperform traditional sinusoidal encodings in preliminary experiments, potentially broadening their application scope.

Contribution

Introduces new periodic functions for positional encoding that differ from sinusoidal functions and demonstrate improved performance in initial tests.

Findings

Alternative functions outperform sinusoidal encoding in experiments

Proposed functions retain key properties of sinusoidal functions

Potential for wider application in transformer architectures

Abstract

A key module in neural transformer-based deep architectures is positional encoding. This module enables a suitable way to encode positional information as input for transformer neural layers. This success has been rooted in the use of sinusoidal functions of various frequencies, in order to capture recurrent patterns of differing typical periods. In this work, an alternative set of periodic functions is proposed for positional encoding. These functions preserve some key properties of sinusoidal ones, while they depart from them in fundamental ways. Some tentative experiments are reported, where the original sinusoidal version is substantially outperformed. This strongly suggests that the alternative functions may have a wider use in other transformer architectures.