Information-Theoretic Generalization Bounds for Sequential Decision Making

TL;DR

This paper develops a new information-theoretic framework for analyzing generalization in sequential decision-making, extending existing bounds to settings like online learning and bandits.

Contribution

It introduces a sequential supersample framework and sequential CMI bounds that handle adaptive data revealing and causal learner trajectories.

Findings

Sequential CMI bounds control generalization gap in adaptive settings.

Bernstein-type refinement achieves faster convergence rates.

Framework applies to online learning, streaming active learning, and bandits.

Abstract

Information-theoretic generalization bounds based on the supersample construction are a central tool for algorithm-dependent generalization analysis in the batch i.i.d.~setting. However, existing supersample conditional mutual information (CMI) bounds do not directly apply to sequential decision-making problems such as online learning, streaming active learning, and bandits, where data are revealed adaptively and the learner evolves along a causal trajectory. To address this limitation, we develop a sequential supersample framework that separates the learner filtration from a proof-side enlargement used for ghost-coordinate comparisons. Under a row-wise exchangeability assumption, the sequential generalization gap is controlled by sequential CMI, a sum of roundwise selector--loss information terms. We also establish a Bernstein-type refinement that yields faster rates under suitable variance conditions. The selector-SCMI proof strategy applies to online learning, streaming active learning with importance weighting, and stochastic multi-armed bandits.