CoPE: A Lightweight Complex Positional Encoding

TL;DR

CoPE introduces a lightweight complex-valued positional encoding for transformers, encoding content and position simultaneously, improving performance and efficiency on benchmarks like GLUE.

Contribution

The paper proposes CoPE, a novel complex-valued positional encoding that captures content and position, with phase-aware attention, enhancing transformer performance and efficiency.

Findings

Outperforms RoPE, Sinusoidal, and Learned encodings on GLUE

Does not exhibit long-term decay in positional information

Compatible with linear attention methods

Abstract

Recent studies have demonstrated the effectiveness of position encoding in transformer architectures. By incorporating positional information, this approach provides essential guidance for modeling dependencies between elements across different sequence positions. We introduce CoPE (a lightweight Complex Positional Encoding), a novel architecture that leverages complex-valued encoding to encode both content and positional information. Our approach replaces traditional positional encodings with complex embeddings where the real part captures semantic content and the imaginary part encodes positional information. We introduce phase-aware attention in the first layer of the transformer model to capture position-dependent patterns, followed by standard attention layers for higher-levels. We show that CoPE doesn't exhibit long term decay and is compatible with linear attention. Experimental evaluation on the GLUE benchmark suggest that our approach achieves superior performance with less computational complexity, compared to RoPE, Sinusoidal and Learned positional encodings.