Bridging the Training-Inference Gap in LLMs by Leveraging Self-Generated Tokens

TL;DR

This paper introduces two training strategies for language models that leverage self-generated tokens to better align training and inference behaviors, improving performance in various NLP tasks.

Contribution

It proposes Batch-Scheduled Sampling and Reference-Answer-based Correction methods that incorporate self-generated tokens during training to reduce the training-inference discrepancy in LLMs.

Findings

Improved performance on summarization tasks

Enhanced accuracy in question-answering tasks

Effective self-correction capabilities demonstrated

Abstract

Language models are often trained to maximize the likelihood of the next token given past tokens in the training dataset. However, during inference time, they are utilized differently, generating text sequentially and auto-regressively by using previously generated tokens as input to predict the next one. Marginal differences in predictions at each step can cascade over successive steps, resulting in different distributions from what the models were trained for and potentially leading to unpredictable behavior. This paper proposes two simple approaches based on model own generation to address this discrepancy between the training and inference time. Our first approach is Batch-Scheduled Sampling, where, during training, we stochastically choose between the ground-truth token from the dataset and the model's own generated token as input to predict the next token. This is done in an offline manner, modifying the context window by interleaving ground-truth tokens with those generated by the model. Our second approach is Reference-Answer-based Correction, where we explicitly incorporate a self-correction capability into the model during training. This enables the model to effectively self-correct the gaps between the generated sequences and the ground truth data without relying on an external oracle model. By incorporating our proposed strategies during training, we have observed an overall improvement in performance compared to baseline methods, as demonstrated by our extensive experiments using summarization, general question-answering, and math question-answering tasks.