Characterizing $L_2$Boosting

TL;DR

This paper provides a detailed analysis of $L_2$Boosting for linear regression, revealing its solution path, unique cycling behavior, and limitations in correlated data, along with a proposed augmentation method to improve its performance.

Contribution

It derives an exact formula for $L_2$Boosting's steps, characterizes its solution path and cycling behavior, and introduces a data augmentation technique to address issues in correlated problems.

Findings

Exact expression for number of steps in $L_2$Boosting

Identification of active set cycling behavior

Data augmentation improves performance in correlated data

Abstract

We consider $L_2$Boosting, a special case of Friedman's generic boosting algorithm applied to linear regression under $L_2$-loss. We study $L_2$Boosting for an arbitrary regularization parameter and derive an exact closed form expression for the number of steps taken along a fixed coordinate direction. This relationship is used to describe $L_2$Boosting's solution path, to describe new tools for studying its path, and to characterize some of the algorithm's unique properties, including active set cycling, a property where the algorithm spends lengthy periods of time cycling between the same coordinates when the regularization parameter is arbitrarily small. Our fixed descent analysis also reveals a repressible condition that limits the effectiveness of $L_2$Boosting in correlated problems by preventing desirable variables from entering the solution path. As a simple remedy, a data augmentation method similar to that used for the elastic net is used to introduce $L_2$-penalization and is shown, in combination with decorrelation, to reverse the repressible condition and circumvents $L_2$Boosting's deficiencies in correlated problems. In itself, this presents a new explanation for why the elastic net is successful in correlated problems and why methods like LAR and lasso can perform poorly in such settings.