Online Gradient Boosting

TL;DR

This paper develops online gradient boosting algorithms that transform weak online learners into strong ones for regression, extending boosting theory to the online learning setting with novel algorithms and optimality results.

Contribution

It introduces online gradient boosting algorithms that generalize boosting to online regression, including a simpler optimal method for convex hulls.

Findings

Proposes an online gradient boosting algorithm for regression.

Provides a simpler, optimal boosting algorithm for convex hulls.

Extends boosting theory to online learning with new algorithms.

Abstract

We extend the theory of boosting for regression problems to the online learning setting. Generalizing from the batch setting for boosting, the notion of a weak learning algorithm is modeled as an online learning algorithm with linear loss functions that competes with a base class of regression functions, while a strong learning algorithm is an online learning algorithm with convex loss functions that competes with a larger class of regression functions. Our main result is an online gradient boosting algorithm which converts a weak online learning algorithm into a strong one where the larger class of functions is the linear span of the base class. We also give a simpler boosting algorithm that converts a weak online learning algorithm into a strong one where the larger class of functions is the convex hull of the base class, and prove its optimality.