Penalization-induced shrinking without rotation in high dimensional GLM regression: a cavity analysis

TL;DR

This paper introduces a generalized ridge penalization method for high-dimensional GLM regression that prevents unwanted rotation of estimated parameters, with analytical insights and validation through simulations.

Contribution

It proposes a novel penalization approach that eliminates rotation in high-dimensional GLMs and analyzes its asymptotic properties using the cavity method.

Findings

The generalized ridge penalization removes rotation effects.

The method's asymptotic behavior is characterized analytically.

Simulation results confirm theoretical predictions.

Abstract

In high dimensional regression, where the number of covariates is of the order of the number of observations, ridge penalization is often used as a remedy against overfitting. Unfortunately, for correlated covariates such regularisation typically induces in generalized linear models not only shrinking of the estimated parameter vector, but also an unwanted \emph{rotation} relative to the true vector. We show analytically how this problem can be removed by using a generalization of ridge penalization, and we analyse the asymptotic properties of the corresponding estimators in the high dimensional regime, using the cavity method. Our results also provide a quantitative rationale for tuning the parameter that controlling the amount of shrinking. We compare our theoretical predictions with simulated data and find excellent agreement.