Optimal and Adaptive Algorithms for Online Boosting

TL;DR

This paper introduces two online boosting algorithms, one optimal and one adaptive, that convert weak online learners into strong ones, with theoretical guarantees and experimental validation.

Contribution

It presents a novel definition of weak online learnability and develops the first optimal online boosting algorithm, along with an adaptive variant.

Findings

The optimal algorithm matches lower bounds in sample complexity.

The adaptive algorithm is parameter-free but not optimal.

Experimental results validate theoretical claims.

Abstract

We study online boosting, the task of converting any weak online learner into a strong online learner. Based on a novel and natural definition of weak online learnability, we develop two online boosting algorithms. The first algorithm is an online version of boost-by-majority. By proving a matching lower bound, we show that this algorithm is essentially optimal in terms of the number of weak learners and the sample complexity needed to achieve a specified accuracy. This optimal algorithm is not adaptive however. Using tools from online loss minimization, we derive an adaptive online boosting algorithm that is also parameter-free, but not optimal. Both algorithms work with base learners that can handle example importance weights directly, as well as by rejection sampling examples with probability defined by the booster. Results are complemented with an extensive experimental study.