Tail-Aware Information-Theoretic Generalization for RLHF and SGLD

TL;DR

This paper introduces a tail-aware information-theoretic framework for analyzing generalization in settings with heavy-tailed data, extending classical bounds to sub-Weibull distributions.

Contribution

It develops a novel tail-dependent divergence and establishes sharp inequalities for sub-Weibull processes, enabling new generalization bounds in heavy-tailed scenarios.

Findings

Provides PAC-Bayes bounds for heavy-tailed data

Derives a chaining inequality based on Rényi mutual information

Applies framework to RLHF and SGLD with heavy-tailed noise

Abstract

Classical information-theoretic generalization bounds typically control the generalization gap through KL-based mutual information and therefore rely on boundedness or sub-Gaussian tails via the moment generating function (MGF). In many modern pipelines, such as robust learning, RLHF, and stochastic optimization, losses and rewards can be heavy-tailed, and MGFs may not exist, rendering KL-based tools ineffective. We develop a tail-dependent information-theoretic framework for sub-Weibull data, where the tail parameter $\theta$ controls the tail heaviness: $\theta=2$ corresponds to sub-Gaussian, $\theta=1$ to sub-exponential, and $0<\theta<1$ to genuinely heavy tails. Our key technical ingredient is a decorrelation lemma that bounds change-of-measure expectations using a shifted-log $f_\theta$-divergence, which admits explicit comparisons to R\'enyi divergence without MGF arguments. On the empirical-process side, we establish sharp maximal inequalities and a Dudley-type chaining bound for sub-Weibull processes with tail index $\theta$, with complexity scaling as $\log^{1/\theta}$ and entropy$^{1/\theta}$. These tools yield expected and high-probability PAC-Bayes generalization bounds, as well as an information-theoretic chaining inequality based on multiscale R\'enyi mutual information. We illustrate the consequences in R\'enyi-regularized RLHF under heavy-tailed rewards and in stochastic gradient Langevin dynamics with heavy-tailed gradient noise.