Increasing transformer token length with a Maximum Entropy Principle Method

TL;DR

This paper introduces three methods based on the Maximum Entropy Principle to extend the token processing length of transformers from T to 2T, aiming to reduce computational overhead while maintaining autoregressive capabilities.

Contribution

The paper proposes novel constraint-based methods that increase transformer token length linearly, using an intermediate step guided by the Maximum Entropy Principle, which is a new approach in this context.

Findings

Extend autoregressive length from T to 2T tokens

Methods are faster than standard approaches despite added overhead

Maintain autoregressive properties with increased token length

Abstract

Transformers suffer from the computational overhead of their quadratic dependence on the length of sequences processed. We present three methods, all adding an intermediate step between training and inference/generation, which extend the autoregressive length of transformers. All rely on a Maximum Entropy Principle (MEP) whereby entropy is maximized in the presence of suitable constraints, accounted for by use of Lagrange Multipliers. These constraint methods extend the autoregressive character from T to 2T tokens in a linear-with-T fashion. There is overhead associated with this added step, but they should still be faster than the standard methods.