Safe preselection in lasso-type problems by cross-validation freezing

TL;DR

This paper introduces a novel 'freezing' property in cross-validation curves that enables safe preselection of variables in high-dimensional penalized regression, improving efficiency and applicability to ultra high-dimensional data.

Contribution

The paper identifies and characterizes the 'freezing' property in cross-validation curves, providing a safe method for variable preselection in lasso-type problems.

Findings

Freezing allows safe variable preselection in high-dimensional data.

Ranking predictors by univariate correlation often leads to early freezing.

The method is applicable to GWAS and microarray data.

Abstract

We propose a new approach to safe variable preselection in high-dimensional penalized regression, such as the lasso. Preselection - to start with a manageable set of covariates - has often been implemented without clear appreciation of its potential bias. Based on sequential implementation of the lasso with increasing lists of predictors, we find a new property of the set of corresponding cross-validation curves, a pattern that we call freezing. It allows to determine a subset of covariates with which we reach the same lasso solution as would be obtained using the full set of covariates. Freezing has not been characterized before and is different from recently discussed safe rules for discarding predictors. We demonstrate by simulation that ranking predictors by their univariate correlation with the outcome, leads in a majority of cases to early freezing, giving a safe and efficient way of focusing the lasso analysis on a smaller and manageable number of predictors. We illustrate the applicability of our strategy in the context of a GWAS analysis and on microarray genomic data. Freezing offers great potential for extending the applicability of penalized regressions to ultra highdimensional data sets. Its applicability is not limited to the standard lasso but is a generic property of many penalized approaches.