Generalization of Hamiltonian algorithms

TL;DR

This paper establishes generalization bounds for stochastic learning algorithms, including Gibbs and PAC-Bayesian methods, under conditions of absolute continuity and subgaussian concentration of the distribution.

Contribution

It provides new theoretical generalization results applicable to a broad class of stochastic algorithms with data-dependent priors.

Findings

Bounds for Gibbs algorithms derived

Generalization results for randomized stable algorithms

PAC-Bayesian bounds with data-dependent priors established

Abstract

The paper proves generalization results for a class of stochastic learning algorithms. The method applies whenever the algorithm generates an absolutely continuous distribution relative to some a-priori measure and the Radon Nikodym derivative has subgaussian concentration. Applications are bounds for the Gibbs algorithm and randomizations of stable deterministic algorithms as well as PAC-Bayesian bounds with data-dependent priors.