Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure

TL;DR

This paper introduces position coupling, a method that embeds task structure into positional encoding, enabling Transformers to generalize arithmetic tasks like addition to much longer sequences than trained on.

Contribution

The paper proposes position coupling, a novel positional encoding technique that improves length generalization in Transformers for arithmetic and other algorithmic tasks.

Findings

Models with position coupling generalize to 200-digit addition after training on 30-digit addition.

Theoretically, a 1-layer Transformer with position coupling can solve exponential-length addition.

Position coupling is applicable to other algorithmic tasks like multiplication and 2D tasks.

Abstract

Even for simple arithmetic tasks like integer addition, it is challenging for Transformers to generalize to longer sequences than those encountered during training. To tackle this problem, we propose position coupling, a simple yet effective method that directly embeds the structure of the tasks into the positional encoding of a (decoder-only) Transformer. Taking a departure from the vanilla absolute position mechanism assigning unique position IDs to each of the tokens, we assign the same position IDs to two or more "relevant" tokens; for integer addition tasks, we regard digits of the same significance as in the same position. On the empirical side, we show that with the proposed position coupling, our models trained on 1 to 30-digit additions can generalize up to 200-digit additions (6.67x of the trained length). On the theoretical side, we prove that a 1-layer Transformer with coupled positions can solve the addition task involving exponentially many digits, whereas any 1-layer Transformer without positional information cannot entirely solve it. We also demonstrate that position coupling can be applied to other algorithmic tasks such as Nx2 multiplication and a two-dimensional task.