On Margins and Generalisation for Voting Classifiers

TL;DR

This paper provides new margin-based generalisation bounds for voting classifiers using PAC-Bayes theory, offering state-of-the-art guarantees and extending the margins theory to non-randomised votes.

Contribution

It introduces margin-based generalisation bounds for voting classifiers that apply to non-randomised votes, expanding the theoretical understanding of ensemble methods.

Findings

State-of-the-art generalisation guarantees for voting classifiers

Bounds applicable to non-randomised votes via margins

Extension of margins theory to ensemble classifiers

Abstract

We study the generalisation properties of majority voting on finite ensembles of classifiers, proving margin-based generalisation bounds via the PAC-Bayes theory. These provide state-of-the-art guarantees on a number of classification tasks. Our central results leverage the Dirichlet posteriors studied recently by Zantedeschi et al. [2021] for training voting classifiers; in contrast to that work our bounds apply to non-randomised votes via the use of margins. Our contributions add perspective to the debate on the "margins theory" proposed by Schapire et al. [1998] for the generalisation of ensemble classifiers.