Jensen-Shannon Information Based Characterization of the Generalization Error of Learning Algorithms

TL;DR

This paper introduces a new information-theoretic generalization error bound using Jensen-Shannon information, which can be tighter than existing mutual information bounds and unifies previous bounds.

Contribution

It proposes a novel Jensen-Shannon information-based bound for generalization error that generalizes and improves upon existing mutual information bounds.

Findings

The bound can specialize to previous bounds.

Under certain conditions, the bound is tighter than mutual information bounds.

The approach unifies various existing generalization bounds.

Abstract

Generalization error bounds are critical to understanding the performance of machine learning models. In this work, we propose a new information-theoretic based generalization error upper bound applicable to supervised learning scenarios. We show that our general bound can specialize in various previous bounds. We also show that our general bound can be specialized under some conditions to a new bound involving the Jensen-Shannon information between a random variable modelling the set of training samples and another random variable modelling the hypothesis. We also prove that our bound can be tighter than mutual information-based bounds under some conditions.