Controlling Multiple Errors Simultaneously with a PAC-Bayes Bound

TL;DR

This paper introduces a novel PAC-Bayes bound that simultaneously controls multiple types of errors, providing richer performance guarantees for models in classification and regression beyond traditional scalar metrics.

Contribution

It presents the first PAC-Bayes bound capable of bounding the distribution of multiple error types simultaneously, including complex metrics like the confusion matrix.

Findings

Bound controls multiple error distributions simultaneously

Transformable into a differentiable training objective

Effective for dynamic error severity scenarios

Abstract

Current PAC-Bayes generalisation bounds are restricted to scalar metrics of performance, such as the loss or error rate. However, one ideally wants more information-rich certificates that control the entire distribution of possible outcomes, such as the distribution of the test loss in regression, or the probabilities of different mis-classifications. We provide the first PAC-Bayes bound capable of providing such rich information by bounding the Kullback-Leibler divergence between the empirical and true probabilities of a set of $M$ error types, which can either be discretized loss values for regression, or the elements of the confusion matrix (or a partition thereof) for classification. We transform our bound into a differentiable training objective. Our bound is especially useful in cases where the severity of different mis-classifications may change over time; existing PAC-Bayes bounds can only bound a particular pre-decided weighting of the error types. In contrast our bound implicitly controls all uncountably many weightings simultaneously.