Surprisal-R\'enyi Free Energy

TL;DR

The paper introduces Surprisal-Rényi Free Energy (SRFE), a new divergence measure that interpolates between forward and reverse KL divergences, revealing a variance-based tradeoff and providing a large deviation and MDL interpretation.

Contribution

SRFE is a novel free-energy functional outside the class of f-divergences, unifying and extending the understanding of KL divergences with variance sensitivity and large deviation control.

Findings

SRFE recovers forward and reverse KL as limits.

SRFE reveals a mean-variance tradeoff in divergence regimes.

SRFE controls large deviations of code-lengths with Chernoff bounds.

Abstract

The forward and reverse Kullback-Leibler (KL) divergences arise as limiting objectives in learning and inference yet induce markedly different inductive biases that cannot be explained at the level of expectations alone. In this work, we introduce the Surprisal-R\'enyi Free Energy (SRFE), a log-moment-based functional of the likelihood ratio that lies outside the class of $f$-divergences. We show that SRFE recovers forward and reverse KL divergences as singular endpoint limits and derive local expansions around both limits in which the variance of the log-likelihood ratio appears as a first-order correction. This reveals an explicit mean-variance tradeoff governing departures from KL-dominated regimes. We further establish a Gibbs-type variational characterization of SRFE as the unique minimizer of a weighted sum of KL divergences and prove that SRFE directly controls large deviations of excess code-length via Chernoff-type bounds, yielding a precise Minimum Description Length interpretation. Together, these results identify SRFE as a variance- and tail-sensitive free-energy functional that clarifies the geometric and large-deviation structure underlying forward and reverse KL limits, without unifying or subsuming distinct learning frameworks.