Self-Bounding Majority Vote Learning Algorithms by the Direct Minimization of a Tight PAC-Bayesian C-Bound

TL;DR

This paper introduces scalable algorithms that directly optimize PAC-Bayesian bounds on the C-Bound for majority vote classifiers, leading to accurate predictions with strong theoretical guarantees.

Contribution

It proposes the first algorithms that directly minimize PAC-Bayesian bounds on the C-Bound, improving theoretical guarantees in majority vote learning.

Findings

Algorithms are scalable and based on gradient descent.

Results show accurate predictors with non-vacuous guarantees.

Demonstrates the effectiveness of direct PAC-Bayesian bound optimization.

Abstract

In the PAC-Bayesian literature, the C-Bound refers to an insightful relation between the risk of a majority vote classifier (under the zero-one loss) and the first two moments of its margin (i.e., the expected margin and the voters' diversity). Until now, learning algorithms developed in this framework minimize the empirical version of the C-Bound, instead of explicit PAC-Bayesian generalization bounds. In this paper, by directly optimizing PAC-Bayesian guarantees on the C-Bound, we derive self-bounding majority vote learning algorithms. Moreover, our algorithms based on gradient descent are scalable and lead to accurate predictors paired with non-vacuous guarantees.