# Strengthened Information-theoretic Bounds on the Generalization Error

## TL;DR

This paper develops new information-theoretic bounds on the probability of error events in learning, improving existing bounds by leveraging various dependence measures like mutual information and maximal leakage.

## Contribution

It introduces strengthened bounds on generalization error using multiple information-theoretic metrics, surpassing previous bounds in tightness.

## Key findings

- Mutual information bounds can outperform existing bounds significantly.
- New bounds are demonstrated using lautum information, maximal leakage, and $J_ty$.
- The bounds provide tighter control over overfitting in learning algorithms.

## Abstract

The following problem is considered: given a joint distribution $P_{XY}$ and an event $E$, bound $P_{XY}(E)$ in terms of $P_XP_Y(E)$ (where $P_XP_Y$ is the product of the marginals of $P_{XY}$) and a measure of dependence of $X$ and $Y$. Such bounds have direct applications in the analysis of the generalization error of learning algorithms, where $E$ represents a large error event and the measure of dependence controls the degree of overfitting. Herein, bounds are demonstrated using several information-theoretic metrics, in particular: mutual information, lautum information, maximal leakage, and $J_\infty$. The mutual information bound can outperform comparable bounds in the literature by an arbitrarily large factor.