Arbitrary-Length Generalization for Addition in a Tiny Transformer

TL;DR

This paper presents a new training method for Tiny Transformers that enables arbitrary-length addition generalization, using an autoregressive approach that mimics manual addition, with reproducible results and available code.

Contribution

It introduces a novel autoregressive training technique for Transformers to perform addition on numbers of unseen lengths, enhancing generalization capabilities.

Findings

Transformer generalizes addition to unseen lengths

Method achieves accurate results on large numbers

Reproducible with publicly available code

Abstract

This paper introduces a novel training methodology that enables a Transformer model to generalize the addition of two-digit numbers to numbers with unseen lengths of digits. The proposed approach employs an autoregressive generation technique, processing from right to left, which mimics a common manual method for adding large numbers. To the best of my knowledge, this methodology has not been previously explored in the literature. All results are reproducible, and the corresponding R code is available at github.com/AGPatriota/ALGA-R/.