Information-theoretic Analysis of the Gibbs Algorithm: An Individual Sample Approach

TL;DR

This paper introduces an individual sample approach to analyze the generalization error of the Gibbs algorithm, providing asymptotic equivalence results and tighter bounds through information-theoretic measures.

Contribution

It establishes the asymptotic equivalence between sample-wise and dataset-wise symmetrized KL information for the Gibbs algorithm, with explicit non-asymptotic bounds.

Findings

Asymptotic equivalence of information measures

Explicit non-asymptotic bounds for the gap

Tighter generalization error bounds

Abstract

Recent progress has shown that the generalization error of the Gibbs algorithm can be exactly characterized using the symmetrized KL information between the learned hypothesis and the entire training dataset. However, evaluating such a characterization is cumbersome, as it involves a high-dimensional information measure. In this paper, we address this issue by considering individual sample information measures within the Gibbs algorithm. Our main contribution lies in establishing the asymptotic equivalence between the sum of symmetrized KL information between the output hypothesis and individual samples and that between the hypothesis and the entire dataset. We prove this by providing explicit expressions for the gap between these measures in the non-asymptotic regime. Additionally, we characterize the asymptotic behavior of various information measures in the context of the Gibbs algorithm, leading to tighter generalization error bounds. An illustrative example is provided to verify our theoretical results, demonstrating our analysis holds in broader settings.