Tighter Information-Theoretic Generalization Bounds via a Novel Class of Change of Measure Inequalities

TL;DR

This paper introduces a new, unified framework for change of measure inequalities that yields tighter bounds and improves analysis in learning theory, privacy, and information theory.

Contribution

The authors develop novel, elementary change of measure inequalities based on the data processing inequality, applicable to various information measures, enhancing theoretical guarantees.

Findings

Derived tighter bounds for generalization error and privacy guarantees.

Unified framework simplifies and strengthens existing theoretical results.

Applicable to multiple divergence measures including f-divergences and Rényi divergence.

Abstract

Change of measure inequalities translate divergences between probability measures into explicit bounds on event probabilities, and play an important role in deriving probabilistic guarantees in learning theory, information theory, and statistics. We propose novel change of measure inequalities via a unified framework based on the data processing inequality, which is surprisingly elementary yet powerful enough to yield novel, tighter inequalities. We provide change of measure inequalities in terms of a broad family of information measures, including $f$-divergences (with Kullback-Leibler divergence and $\chi^2$-divergence as special cases), R\'enyi divergence, and $\alpha$-mutual information (with maximal leakage as a special case). We apply these results to generalization error analysis, PAC-Bayesian theory, differential privacy, and data memorization, obtaining stronger guarantees while recovering best-known results through simplified analyses.