A Generalized Stacking for Implementing Ensembles of Gradient Boosting Machines

TL;DR

This paper introduces a generalized stacking approach to create ensembles of gradient boosting machines, leveraging differentiable meta-models like linear regression and neural networks to improve regression performance.

Contribution

It proposes a flexible ensemble method using stacking with differentiable meta-models, extending beyond linear models to neural networks for gradient boosting ensembles.

Findings

Effective ensemble construction demonstrated through numerical examples.

Flexible extension to neural networks enhances ensemble modeling capabilities.

Improved regression performance over single gradient boosting models.

Abstract

The gradient boosting machine is one of the powerful tools for solving regression problems. In order to cope with its shortcomings, an approach for constructing ensembles of gradient boosting models is proposed. The main idea behind the approach is to use the stacking algorithm in order to learn a second-level meta-model which can be regarded as a model for implementing various ensembles of gradient boosting models. First, the linear regression of the gradient boosting models is considered as a simplest realization of the meta-model under condition that the linear model is differentiable with respect to its coefficients (weights). Then it is shown that the proposed approach can be simply extended on arbitrary differentiable combination models, for example, on neural networks which are differentiable and can implement arbitrary functions of gradient boosting models. Various numerical examples illustrate the proposed approach.