Perturbation is All You Need for Extrapolating Language Models

TL;DR

This paper proposes a perturbation-based training framework for large language models that enhances their ability to predict sequences outside the training data support, with theoretical and empirical validation.

Contribution

It introduces a novel perturbation approach for language modeling that improves extrapolability and provides a rigorous theoretical foundation for out-of-support predictions.

Findings

Improved out-of-support prediction accuracy.

Maintains competitive in-support performance.

Theoretically grounded in extrapolability analysis.

Abstract

We introduce a simple yet powerful framework for training large language models. In contrast to the standard autoregressive next-token prediction based on an exact prefix, we propose a perturbation-based procedure that first transforms the prefix into a semantic neighbor and then conditions on this perturbed variant for next-token prediction. This yields a hierarchical model with a pre-post-additive noise structure. Within this framework, we develop a rigorous theory of extrapolability, namely, the capacity of a model class to make reliable predictions for token sequences that lie outside the empirical support of the training corpus. We evaluate the finite-sample performance of the proposed procedure using both synthetic and real-world language data. Results show that the proposed method consistently improves out-of-support prediction while maintaining competitive in-support performance, demonstrating that perturbation offers a practical route to language modeling.