Improved Margin Generalization Bounds for Voting Classifiers

TL;DR

This paper introduces a new margin-based generalization bound for voting classifiers, improving theoretical guarantees for boosting algorithms and proposing an optimal weak-to-strong learner with minimal expected error.

Contribution

It presents a tighter margin-based generalization bound and derives an optimal Majority-of-3 classifier that matches theoretical lower bounds.

Findings

Tighter generalization bounds for boosting algorithms.

An optimal Majority-of-3 classifier with minimal expected error.

Improved theoretical understanding of voting classifier performance.

Abstract

In this paper we establish a new margin-based generalization bound for voting classifiers, refining existing results and yielding tighter generalization guarantees for widely used boosting algorithms such as AdaBoost (Freund and Schapire, 1997). Furthermore, the new margin-based generalization bound enables the derivation of an optimal weak-to-strong learner: a Majority-of-3 large-margin classifiers with an expected error matching the theoretical lower bound. This result provides a more natural alternative to the Majority-of-5 algorithm by (H{\o}gsgaard et al., 2024), and matches the Majority-of-3 result by (Aden-Ali et al., 2024) for the realizable prediction model.