Individually Conditional Individual Mutual Information Bound on Generalization Error

TL;DR

This paper introduces a new information-theoretic bound on generalization error that improves upon previous bounds by reducing conditioning terms, supported by a novel decoupling lemma and comparative analysis.

Contribution

It proposes a novel bound on generalization error that overcomes limitations of prior bounds by reducing conditioning, with a new decoupling lemma for analysis.

Findings

The new bound outperforms previous bounds in Gaussian settings.

A conditional decoupling lemma is established to support the analysis.

The paper provides a meaningful comparison among different information-theoretic bounds.

Abstract

We propose a new information-theoretic bound on generalization error based on a combination of the error decomposition technique of Bu et al. and the conditional mutual information (CMI) construction of Steinke and Zakynthinou. In a previous work, Haghifam et al. proposed a different bound combining the two aforementioned techniques, which we refer to as the conditional individual mutual information (CIMI) bound. However, in a simple Gaussian setting, both the CMI and the CIMI bounds are order-wise worse than that by Bu et al.. This observation motivated us to propose the new bound, which overcomes this issue by reducing the conditioning terms in the conditional mutual information. In the process of establishing this bound, a conditional decoupling lemma is established, which also leads to a meaningful dichotomy and comparison among these information-theoretic bounds.