Deselection of Base-Learners for Statistical Boosting -- with an Application to Distributional Regression

TL;DR

This paper introduces a new deselection procedure for component-wise gradient boosting to improve variable selection, especially in low-dimensional data, demonstrated through a distributional regression application in a health study.

Contribution

The paper proposes a novel method to deselect less important base-learners in boosting, reducing false positives and improving model interpretability.

Findings

Enhanced variable selection accuracy in low-dimensional data

Reduced inclusion of false positive variables

Improved prediction performance compared to existing methods

Abstract

We present a new procedure for enhanced variable selection for component-wise gradient boosting. Statistical boosting is a computational approach that emerged from machine learning, which allows to fit regression models in the presence of high-dimensional data. Furthermore, the algorithm can lead to data-driven variable selection. In practice, however, the final models typically tend to include too many variables in some situations. This occurs particularly for low-dimensional data (p<n), where we observe a slow overfitting behavior of boosting. As a result, more variables get included into the final model without altering the prediction accuracy. Many of these false positives are incorporated with a small coefficient and therefore have a small impact, but lead to a larger model. We try to overcome this issue by giving the algorithm the chance to deselect base-learners with minor importance. We analyze the impact of the new approach on variable selection and prediction performance in comparison to alternative methods including boosting with earlier stopping as well as twin boosting. We illustrate our approach with data of an ongoing cohort study for chronic kidney disease patients, where the most influential predictors for the health-related quality of life measure are selected in a distributional regression approach based on beta regression.