Exhuming nonnegative garrote from oblivion using suitable initial estimates- illustration in low and high-dimensional real data

TL;DR

This paper revisits the nonnegative garrote (NNG) method, demonstrating its applicability in high-dimensional data using suitable initial estimates, and compares its performance with other variable selection techniques.

Contribution

It shows that replacing initial estimates in NNG enables its use in high-dimensional settings, offering simpler, interpretable models with competitive prediction accuracy.

Findings

NNG can be effectively applied in high-dimensional data with appropriate initial estimates.

Replacing OLS with ridge estimates improves model simplicity in high multicollinearity scenarios.

NNG performs comparably to lasso and other methods in variable selection and prediction.

Abstract

The nonnegative garrote (NNG) is among the first approaches that combine variable selection and shrinkage of regression estimates. When more than the derivation of a predictor is of interest, NNG has some conceptual advantages over the popular lasso. Nevertheless, NNG has received little attention. The original NNG relies on least-squares (OLS) estimates, which are highly variable in data with a high degree of multicollinearity (HDM) and do not exist in high-dimensional data (HDD). This might be the reason that NNG is not used in such data. Alternative initial estimates have been proposed but hardly used in practice. Analyzing three structurally different data sets, we demonstrated that NNG can also be applied in HDM and HDD and compared its performance with the lasso, adaptive lasso, relaxed lasso, and best subset selection in terms of variables selected, regression estimates, and prediction. Replacing OLS by ridge initial estimates in HDM and lasso initial estimates in HDD helped NNG select simpler models than competing approaches without much increase in prediction errors. Simpler models are easier to interpret, an important issue for descriptive modelling. Based on the limited experience from three datasets, we assume that the NNG can be a suitable alternative to the lasso and its extensions. Neutral comparison simulation studies are needed to better understand the properties of variable selection methods, compare them and derive guidance for practice.