A Theoretical Lens for RL-Tuned Language Models via Energy-Based Models

TL;DR

This paper provides a theoretical framework for understanding RL-tuned language models by analyzing their energy-based model structure, revealing convergence properties and explaining empirical entropy-accuracy trade-offs.

Contribution

It introduces a unified variational analysis of RL-tuned LLMs using energy-based models, establishing convergence guarantees and insights into their training dynamics.

Findings

Monotonic KL convergence to high-quality distributions

Bounded hitting times to better states

Explanation of entropy-accuracy trade-offs

Abstract

Large language models (LLMs) trained via KL-regularized reinforcement learning demonstrate strong instruction following, self-correction, and reasoning abilities. Yet their theoretical underpinnings remain limited. We exploit the closed-form energy-based model (EBM) structure of the optimal KL-regularized policy to provide a unified variational analysis of LLMs. For instruction-tuned models, under natural assumptions on reward potentials and pretraining symmetry, we prove that the transition kernel satisfies detailed balance with respect to a scalar potential encoding response quality. This yields monotonic KL convergence to a high-quality stationary distribution, bounded hitting times to superior states, and exponential mixing governed by the spectral gap. For reasoning models trained with verifiable rewards (RLVR), we show the objective is equivalent to expected KL minimization toward an optimal reasoning distribution, with the suboptimality gap reducing to the Bernoulli KL between target and current accuracies along the natural gradient flow. This helps explain empirical entropy-accuracy trade-offs.