Adaptive Step-Length Selection in Gradient Boosting for Generalized Additive Models for Location, Scale and Shape

TL;DR

This paper introduces an adaptive step-length method for gradient boosting in GAMLSS, improving model fitting and computational efficiency, especially in complex scenarios with multiple predictors and large variance.

Contribution

It proposes an adaptive step-length approach for GAMLSS boosting, including a semi-analytical form for Gaussian models, enhancing efficiency and balancing predictor updates.

Findings

Competitive performance against penalized maximum likelihood.

Effective in large variance and high-dimensional settings.

Applicable to other models with multiple predictors.

Abstract

Tuning of model-based boosting algorithms relies mainly on the number of iterations, while the step-length is fixed at a predefined value. For complex models with several predictors such as Generalized Additive Models for Location, Scale and Shape (GAMLSS), imbalanced updates of predictors, where some distribution parameters are updated more frequently than others, can be a problem that prevents some submodels to be appropriately fitted within a limited number of boosting iterations. We propose an approach using adaptive step-length (ASL) determination within a non-cyclical boosting algorithm for GAMLSS to prevent such imbalance. Moreover, for the important special case of the Gaussian distribution, we discuss properties of the ASL and derive a semi-analytical form of the ASL that avoids manual selection of the search interval and numerical optimization to find the optimal step-length, and consequently improves computational efficiency. We show competitive behavior of the proposed approaches compared to penalized maximum likelihood and boosting with a fixed step-length for GAMLSS models in two simulations and two applications, in particular for cases of large variance and/or more variables than observations. In addition, the idea of the ASL is also applicable to other models with more than one predictor like zero-inflated count model, and brings up insights into the choice of the reasonable defaults for the step-length in simpler special case of (Gaussian) additive models.