Think before you speak: Training Language Models With Pause Tokens

TL;DR

This paper introduces pause tokens to allow language models to perform extra computation before generating each token, improving performance on various tasks by delaying output until additional processing is completed.

Contribution

The paper proposes a novel pause token mechanism enabling models to manipulate extra hidden vectors before output, enhancing task performance through inference-time delays.

Findings

Significant improvements on question-answering and reasoning tasks.

Effective delay-based training enhances model accuracy.

Demonstrated gains on multiple downstream tasks.

Abstract

Language models generate responses by producing a series of tokens in immediate succession: the $(K+1)^{th}$ token is an outcome of manipulating $K$ hidden vectors per layer, one vector per preceding token. What if instead we were to let the model manipulate say, $K+10$ hidden vectors, before it outputs the $(K+1)^{th}$ token? We operationalize this idea by performing training and inference on language models with a (learnable) $\textit{pause}$ token, a sequence of which is appended to the input prefix. We then delay extracting the model's outputs until the last pause token is seen, thereby allowing the model to process extra computation before committing to an answer. We empirically evaluate $\textit{pause-training}$ on decoder-only models of 1B and 130M parameters with causal pretraining on C4, and on downstream tasks covering reasoning, question-answering, general understanding and fact recall. Our main finding is that inference-time delays show gains when the model is both pre-trained and finetuned with delays. For the 1B model, we witness gains on 8 of 9 tasks, most prominently, a gain of $18\%$ EM score on the QA task of SQuAD, $8\%$ on CommonSenseQA and $1\%$ accuracy on the reasoning task of GSM8k. Our work raises a range of conceptual and practical future research questions on making delayed next-token prediction a widely applicable new paradigm.