A Note on the Efficient Evaluation of PAC-Bayes Bounds

TL;DR

This paper introduces a computationally efficient method for evaluating PAC-Bayes bounds, significantly reducing the dataset passes needed for risk estimation in machine learning models.

Contribution

It proposes a general approach that cuts down the computational cost of PAC-Bayes risk bounds estimation, making it more practical for large datasets.

Findings

Reduces dataset passes for risk estimation by a factor proportional to dataset size

Provides a general method applicable to various PAC-Bayes bounds

Achieves significant computational savings in risk certification

Abstract

When utilising PAC-Bayes theory for risk certification, it is usually necessary to estimate and bound the Gibbs risk of the PAC-Bayes posterior. Many works in the literature employ a method for this which requires a large number of passes of the dataset, incurring high computational cost. This manuscript presents a very general alternative which makes computational savings on the order of the dataset size.