Sharp Trade-Offs in High-Dimensional Inference via 2-Level SLOPE

TL;DR

This paper introduces 2-level SLOPE, a simplified yet effective variant of SLOPE for high-dimensional linear regression, offering improved statistical properties and computational efficiency, especially in correlated or noisy data scenarios.

Contribution

The paper proposes 2-level SLOPE with only three hyperparameters, providing a sharp theoretical characterization of the TPP-FDP trade-off and demonstrating practical advantages over existing methods.

Findings

2-level SLOPE maintains SLOPE's advantages like improved MSE and power.

It offers computational benefits by reducing hyperparameter complexity.

Empirical results show robustness in correlated, noisy, and non-sparse settings.

Abstract

Among techniques for high-dimensional linear regression, Sorted L-One Penalized Estimation (SLOPE) generalizes the LASSO via an adaptive $l_1$ regularization that applies heavier penalties to larger coefficients in the model. To achieve such adaptivity, SLOPE requires the specification of a complex hierarchy of penalties, i.e., a monotone penalty sequence in $R^p$, in contrast to a single penalty scalar for LASSO. Tuning this sequence when $p$ is large poses a challenge, as brute force search over a grid of values is computationally prohibitive. In this work, we study the 2-level SLOPE, an important subclass of SLOPE, with only three hyperparameters. We demonstrate both empirically and analytically that 2-level SLOPE not only preserves the advantages of general SLOPE -- such as improved mean squared error and overcoming the Donoho-Tanner power limit -- but also exhibits computational benefits by reducing the penalty hyperparameter space. In particular, we prove that 2-level SLOPE admits a sharp, theoretically tight characterization of the trade-off between true positive proportion (TPP) and false discovery proportion (FDP), contrasting with general SLOPE where only upper and lower bounds are known. Empirical evaluations further underscore the effectiveness of 2-level SLOPE in settings where predictors exhibit high correlation, when the noise is large, or when the underlying signal is not sparse. Our results suggest that 2-level SLOPE offers a robust, scalable alternative to both LASSO and general SLOPE, making it particularly suited for practical high-dimensional data analysis.