A Framework for Bounding Deterministic Risk with PAC-Bayes: Applications to Majority Votes

TL;DR

This paper introduces a unified PAC-Bayes framework that provides deterministic risk guarantees, enabling practical deployment of single hypotheses with strong generalization bounds, especially for majority vote classifiers.

Contribution

It develops a method to derive deterministic risk guarantees from stochastic PAC-Bayesian bounds, including a general oracle bound and a specialized majority vote bound.

Findings

Outperforms popular baselines in generalization bounds for deterministic classifiers

Provides a unified approach to extract deterministic guarantees from PAC-Bayes

Empirically demonstrates up to a twofold improvement in bounds

Abstract

PAC-Bayes is a popular and efficient framework for obtaining generalization guarantees in situations involving uncountable hypothesis spaces. Unfortunately, in its classical formulation, it only provides guarantees on the expected risk of a randomly sampled hypothesis. This requires stochastic predictions at test time, making PAC-Bayes unusable in many practical situations where a single deterministic hypothesis must be deployed. We propose a unified framework to extract guarantees holding for a single hypothesis from stochastic PAC-Bayesian guarantees. We present a general oracle bound and derive from it a numerical bound and a specialization to majority vote. We empirically show that our approach consistently outperforms popular baselines (by up to a factor of 2) when it comes to generalization bounds on deterministic classifiers.