A large sample theory for infinitesimal gradient boosting

TL;DR

This paper develops a theoretical framework for infinitesimal gradient boosting, analyzing its large sample behavior and demonstrating convergence to a deterministic process that improves test error over time.

Contribution

It introduces a large sample theory for infinitesimal gradient boosting, characterizing its population limit via differential equations and analyzing its convergence and properties.

Findings

Test error decreases over time in the population limit

The dynamics converge to a deterministic process

Long-term behavior shows continued improvement

Abstract

Infinitesimal gradient boosting (Dombry and Duchamps, 2021) is defined as the vanishing-learning-rate limit of the popular tree-based gradient boosting algorithm from machine learning. It is characterized as the solution of a nonlinear ordinary differential equation in a infinite-dimensional function space where the infinitesimal boosting operator driving the dynamics depends on the training sample. We consider the asymptotic behavior of the model in the large sample limit and prove its convergence to a deterministic process. This population limit is again characterized by a differential equation that depends on the population distribution. We explore some properties of this population limit: we prove that the dynamics makes the test error decrease and we consider its long time behavior.