Reversal Invariance in Autoregressive Language Models

TL;DR

This paper introduces the concept of reversal invariance in autoregressive language models, showing that they are inherently symmetric to text reversal, which may limit their ability to model directional language features.

Contribution

It formalizes reversal invariance as a structural property of CLM, analyzes its implications, and proposes the need for asymmetric objectives to better capture language directionality.

Findings

Models trained on reversed text perform comparably to forward-trained models.

Reversal invariance explains why standard CLM is direction-blind.

Current objectives may fail to encode directional dependencies in language.

Abstract

We formalize a structural property of the causal (autoregressive) language modeling (CLM) objective: reversal invariance. Formally, the next-token prediction loss assigns identical likelihood to a corpus and its reversal, implying that standard CLM pretraining is direction-blind. This symmetry explains why models trained on reversed text can achieve comparable performance to those trained on forward text, despite the inherently time-asymmetric nature of human language and reasoning. We argue that this invariance represents a limitation of current pretraining objectives rather than a benign artifact. If natural language encodes directional dependencies - phonological, morphological, or causal - a symmetric objective may fail to capture them. We therefore propose viewing pretraining through the lens of temporal asymmetry, motivating future work on loss functions and architectures that explicitly model the arrow of language while retaining standard language modeling capacity.