Improving Transformers using Faithful Positional Encoding

TL;DR

This paper introduces a mathematically grounded positional encoding for Transformers that preserves input order information and enhances performance in time-series classification tasks.

Contribution

The paper presents a novel positional encoding method for Transformers, ensuring order information is retained and improving prediction accuracy.

Findings

Systematic performance improvement in time-series classification

Mathematically guaranteed preservation of positional information

Outperforms standard sinusoidal encoding in experiments

Abstract

We propose a new positional encoding method for a neural network architecture called the Transformer. Unlike the standard sinusoidal positional encoding, our approach is based on solid mathematical grounds and has a guarantee of not losing information about the positional order of the input sequence. We show that the new encoding approach systematically improves the prediction performance in the time-series classification task.