Information-Theoretic Bounds on the Moments of the Generalization Error of Learning Algorithms

TL;DR

This paper introduces new information-theoretic bounds on the moments of the generalization error of learning algorithms, providing a refined understanding of their performance and linking to existing bounds.

Contribution

It presents a novel analysis of generalization error moments using information-theoretic bounds, extending to high-probability bounds and improving upon prior results.

Findings

Derived bounds for the moments of the generalization error.

Connected new bounds to existing generalization bounds.

Showed how to construct high-probability bounds from moments.

Abstract

Generalization error bounds are critical to understanding the performance of machine learning models. In this work, building upon a new bound of the expected value of an arbitrary function of the population and empirical risk of a learning algorithm, we offer a more refined analysis of the generalization behaviour of a machine learning models based on a characterization of (bounds) to their generalization error moments. We discuss how the proposed bounds -- which also encompass new bounds to the expected generalization error -- relate to existing bounds in the literature. We also discuss how the proposed generalization error moment bounds can be used to construct new generalization error high-probability bounds.