A Unifying Framework of High-Dimensional Sparse Estimation with Difference-of-Convex (DC) Regularizations

TL;DR

This paper presents a unifying DC framework for high-dimensional sparse estimation, demonstrating that certain stationary solutions have desirable statistical properties, thus bridging optimization and statistical theory.

Contribution

It shows that a subset of directional-stationary solutions in DC-regularized problems possess key statistical properties, unifying various non-convex penalties under a common framework.

Findings

Certain d-stationary solutions are asymptotically consistent and efficient.

The DC framework encompasses many existing non-convex penalties.

Assumptions are weaker or comparable to existing conditions.

Abstract

Under the linear regression framework, we study the variable selection problem when the underlying model is assumed to have a small number of nonzero coefficients (i.e., the underlying linear model is sparse). Non-convex penalties in specific forms are well-studied in the literature for sparse estimation. A recent work \cite{ahn2016difference} has pointed out that nearly all existing non-convex penalties can be represented as difference-of-convex (DC) functions, which can be expressed as the difference of two convex functions, while itself may not be convex. There is a large existing literature on the optimization problems when their objectives and/or constraints involve DC functions. Efficient numerical solutions have been proposed. Under the DC framework, directional-stationary (d-stationary) solutions are considered, and they are usually not unique. In this paper, we show that under some mild conditions, a certain subset of d-stationary solutions in an optimization problem (with a DC objective) has some ideal statistical properties: namely, asymptotic estimation consistency, asymptotic model selection consistency, asymptotic efficiency. The aforementioned properties are the ones that have been proven by many researchers for a range of proposed non-convex penalties in the sparse estimation. Our assumptions are either weaker than or comparable with those conditions that have been adopted in other existing works. This work shows that DC is a nice framework to offer a unified approach to these existing work where non-convex penalty is involved. Our work bridges the communities of optimization and statistics.