The EOS Decision and Length Extrapolation

TL;DR

This paper investigates how the use of an end-of-sequence token affects length extrapolation in neural language models, revealing that models trained without EOS predict better for longer sequences and generalize more effectively.

Contribution

It demonstrates that training models without EOS tokens significantly improves their ability to extrapolate to longer sequences and provides insights into the hidden state dynamics responsible for this.

Findings

-EOS models struggle with length extrapolation, especially for sequences 10x longer than training.

-EOS models underperform on the SCAN length generalization task, with 40% lower accuracy.

Models without EOS better avoid length manifolds and attractors, enhancing generalization.

Abstract

Extrapolation to unseen sequence lengths is a challenge for neural generative models of language. In this work, we characterize the effect on length extrapolation of a modeling decision often overlooked: predicting the end of the generative process through the use of a special end-of-sequence (EOS) vocabulary item. We study an oracle setting - forcing models to generate to the correct sequence length at test time - to compare the length-extrapolative behavior of networks trained to predict EOS (+EOS) with networks not trained to (-EOS). We find that -EOS substantially outperforms +EOS, for example extrapolating well to lengths 10 times longer than those seen at training time in a bracket closing task, as well as achieving a 40% improvement over +EOS in the difficult SCAN dataset length generalization task. By comparing the hidden states and dynamics of -EOS and +EOS models, we observe that +EOS models fail to generalize because they (1) unnecessarily stratify their hidden states by their linear position is a sequence (structures we call length manifolds) or (2) get stuck in clusters (which we refer to as length attractors) once the EOS token is the highest-probability prediction.