Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective

TL;DR

This paper introduces a stochastic thermodynamics framework for autoregressive generative models, enabling entropy production estimation in non-Markovian processes like language models, with applications to GPT-2 and Kalman filters.

Contribution

It develops a general theoretical approach to quantify irreversibility in non-Markovian generative models, including entropy production estimation and decomposition into meaningful components.

Findings

Entropy production can be estimated efficiently from sampled trajectories.

Token-level entropy production is dominated by syntactic artifacts.

Sentence-level entropy production can distinguish causal from non-causal text.

Abstract

Autoregressive generative models -- including Transformers, recurrent neural networks, classical Kalman filters, state space models, and Mamba -- all generate sequences by sampling each output from a deterministic summary of the past, producing genuinely non-Markovian observed processes. We develop a general theoretical framework based on stochastic thermodynamics for this class of architectures and introduce the entropy production, which can be efficiently estimated from sampled trajectories without exponential sampling cost, despite the non-Markovian nature of the observed dynamics. As a proof-of-concept experiment with a large language model (LLM), we evaluate the entropy production for a pre-trained Transformer-based model, GPT-2. We find that the token-level entropy production is dominated by a syntactic artifact, while the sentence-level entropy production may yield a more interpretable signal in comparisons between causally ordered and non-causal text sets. We also demonstrate the framework in the linear Gaussian case, where the model reduces to the Kalman innovation representation and the entropy production admits an analytical expression. We further show that the entropy production decomposes exactly into non-negative per-step contributions in terms of retrospective inference, and each of those terms further splits into information-theoretically meaningful terms: a compression loss and a model mismatch. Our results establish a bridge between stochastic thermodynamics and modern generative models, and provide a starting point for quantifying irreversibility in a broad class of highly non-Markovian processes such as LLMs.