TRBoost: A Generic Gradient Boosting Machine based on Trust-region Method

TL;DR

TRBoost introduces a trust-region based gradient boosting method that balances the flexibility of first-order methods with the performance of second-order algorithms, applicable to arbitrary loss functions without Hessian positivity constraints.

Contribution

This paper presents TRBoost, a novel gradient boosting algorithm using trust-region optimization, enabling application to any loss function while maintaining competitive performance.

Findings

TRBoost is as general as first-order GBMs.

TRBoost achieves performance comparable to second-order GBMs.

Numerical experiments confirm the effectiveness of TRBoost.

Abstract

Gradient Boosting Machines (GBMs) have demonstrated remarkable success in solving diverse problems by utilizing Taylor expansions in functional space. However, achieving a balance between performance and generality has posed a challenge for GBMs. In particular, gradient descent-based GBMs employ the first-order Taylor expansion to ensure applicability to all loss functions, while Newton's method-based GBMs use positive Hessian information to achieve superior performance at the expense of generality. To address this issue, this study proposes a new generic Gradient Boosting Machine called Trust-region Boosting (TRBoost). In each iteration, TRBoost uses a constrained quadratic model to approximate the objective and applies the Trust-region algorithm to solve it and obtain a new learner. Unlike Newton's method-based GBMs, TRBoost does not require the Hessian to be positive definite, thereby allowing it to be applied to arbitrary loss functions while still maintaining competitive performance similar to second-order algorithms. The convergence analysis and numerical experiments conducted in this study confirm that TRBoost is as general as first-order GBMs and yields competitive results compared to second-order GBMs. Overall, TRBoost is a promising approach that balances performance and generality, making it a valuable addition to the toolkit of machine learning practitioners.