$\sigma$-Ridge: group regularized ridge regression via empirical Bayes noise level cross-validation

TL;DR

This paper introduces a group-regularized ridge regression method that uses empirical Bayes and noise level cross-validation to optimally select regularization parameters for feature groups, improving predictive risk in high-dimensional settings.

Contribution

It develops a novel empirical Bayes approach for selecting group-specific regularization parameters in ridge regression, with theoretical risk guarantees and practical efficiency.

Findings

Derives asymptotic risk formulas for group-regularized ridge regression.

Proposes a fast, data-driven method for optimal regularization parameter selection.

Achieves asymptotic optimality comparable to theoretical benchmarks.

Abstract

Features in predictive models are not exchangeable, yet common supervised models treat them as such. Here we study ridge regression when the analyst can partition the features into $K$ groups based on external side-information. For example, in high-throughput biology, features may represent gene expression, protein abundance or clinical data and so each feature group represents a distinct modality. The analyst's goal is to choose optimal regularization parameters $\lambda = (\lambda_1, \dotsc, \lambda_K)$ -- one for each group. In this work, we study the impact of $\lambda$ on the predictive risk of group-regularized ridge regression by deriving limiting risk formulae under a high-dimensional random effects model with $p\asymp n$ as $n \to \infty$. Furthermore, we propose a data-driven method for choosing $\lambda$ that attains the optimal asymptotic risk: The key idea is to interpret the residual noise variance $\sigma^2$, as a regularization parameter to be chosen through cross-validation. An empirical Bayes construction maps the one-dimensional parameter $\sigma$ to the $K$-dimensional vector of regularization parameters, i.e., $\sigma \mapsto \widehat{\lambda}(\sigma)$. Beyond its theoretical optimality, the proposed method is practical and runs as fast as cross-validated ridge regression without feature groups ($K=1$).