Gradient boosting machine with partially randomized decision trees

TL;DR

This paper introduces a novel gradient boosting approach using partially randomized decision trees to improve regression smoothness and reduce computational complexity, demonstrated through extensive numerical experiments on synthetic and real datasets.

Contribution

It proposes integrating partially randomized trees into gradient boosting to address discontinuity issues and enhance efficiency, a novel combination in ensemble learning.

Findings

Reduced regression discontinuities in experiments

Lower computational complexity observed

Effective on both synthetic and real data

Abstract

The gradient boosting machine is a powerful ensemble-based machine learning method for solving regression problems. However, one of the difficulties of its using is a possible discontinuity of the regression function, which arises when regions of training data are not densely covered by training points. In order to overcome this difficulty and to reduce the computational complexity of the gradient boosting machine, we propose to apply the partially randomized trees which can be regarded as a special case of the extremely randomized trees applied to the gradient boosting. The gradient boosting machine with the partially randomized trees is illustrated by means of many numerical examples using synthetic and real data.