Replica Analysis for Ensemble Techniques in Variable Selection

TL;DR

This paper uses the replica method from statistical mechanics to analyze the performance of ensemble techniques like stability selection and knockoffs in high-dimensional variable selection, revealing their relative strengths.

Contribution

It introduces a systematic analytical framework for evaluating ensemble methods in high-dimensional settings using the replica approach.

Findings

dKO outperforms vanilla knockoff and SS

Increasing bootstrap resampling in SS can improve detection power

Analytical insights into ensemble method performance in high dimensions

Abstract

Variable selection is a problem of statistics that aims to find the subset of the $N$-dimensional possible explanatory variables that are truly related to the generation process of the response variable. In high-dimensional setups, where the input dimension $N$ is comparable to the data size $M$, it is difficult to use classic methods based on $p$-values. Therefore, methods based on the ensemble learning are often used. In this review article, we introduce how the performance of these ensemble-based methods can be systematically analyzed using the replica method from statistical mechanics when $N$ and $M$ diverge at the same rate as $N,M\to\infty, M/N\to\alpha\in(0,\infty)$. As a concrete application, we analyze the power of stability selection (SS) and the derandomized knockoff (dKO) with the $\ell_1$-regularized statistics in the high-dimensional linear model. The result indicates that dKO provably outperforms the vanilla knockoff and the standard SS, while increasing the bootstrap resampling rate in SS might further improve the detection power.