Accelerated Componentwise Gradient Boosting using Efficient Data Representation and Momentum-based Optimization

TL;DR

This paper introduces two techniques—feature discretization and Nesterov momentum—to significantly improve the efficiency of componentwise gradient boosting, making it faster and more memory-efficient while preserving interpretability.

Contribution

It proposes novel methods to reduce computational complexity in componentwise boosting, including feature discretization and momentum-based optimization with a hybrid approach.

Findings

Reduced runtime and memory usage demonstrated on multiple datasets.

Maintained state-of-the-art prediction accuracy.

Faster convergence with controlled overfitting.

Abstract

Componentwise boosting (CWB), also known as model-based boosting, is a variant of gradient boosting that builds on additive models as base learners to ensure interpretability. CWB is thus often used in research areas where models are employed as tools to explain relationships in data. One downside of CWB is its computational complexity in terms of memory and runtime. In this paper, we propose two techniques to overcome these issues without losing the properties of CWB: feature discretization of numerical features and incorporating Nesterov momentum into functional gradient descent. As the latter can be prone to early overfitting, we also propose a hybrid approach that prevents a possibly diverging gradient descent routine while ensuring faster convergence. We perform extensive benchmarks on multiple simulated and real-world data sets to demonstrate the improvements in runtime and memory consumption while maintaining state-of-the-art estimation and prediction performance.