Infinitesimal gradient boosting

TL;DR

This paper introduces infinitesimal gradient boosting, a new theoretical framework for understanding gradient boosting as a continuous process, with convergence guarantees and insights into its training dynamics.

Contribution

It formalizes the limit of gradient boosting as a differential equation, introduces a new class of randomized trees, and characterizes the algorithm's convergence in a rigorous mathematical setting.

Findings

Convergence of the stochastic algorithm to a unique solution.

The limiting process is described by a nonlinear ODE in function space.

The method provides a smooth training path with controlled residuals.

Abstract

We define infinitesimal gradient boosting as a limit of the popular tree-based gradient boosting algorithm from machine learning. The limit is considered in the vanishing-learning-rate asymptotic, that is when the learning rate tends to zero and the number of gradient trees is rescaled accordingly. For this purpose, we introduce a new class of randomized regression trees bridging totally randomized trees and Extra Trees and using a softmax distribution for binary splitting. Our main result is the convergence of the associated stochastic algorithm and the characterization of the limiting procedure as the unique solution of a nonlinear ordinary differential equation in a infinite dimensional function space. Infinitesimal gradient boosting defines a smooth path in the space of continuous functions along which the training error decreases, the residuals remain centered and the total variation is well controlled.