On Model Selection Consistency of Lasso for High-Dimensional Ising Models

TL;DR

This paper provides a rigorous theoretical analysis of the model selection consistency of Lasso, with and without post-thresholding, for high-dimensional Ising models on various graph structures, matching prior conjectures and extending understanding.

Contribution

It rigorously proves the model selection consistency of Lasso for high-dimensional Ising models on regular and tree-like graphs, confirming previous non-rigorous conjectures and extending to post-thresholding methods.

Findings

Lasso without post-thresholding is consistent in the paramagnetic phase for RR graphs with sample complexity $n= ilde{O}(d^3\log p)$.

The same consistency result extends to general tree-like graphs under mild conditions.

Lasso with post-thresholding is consistent for general tree-like graphs without additional assumptions.

Abstract

We theoretically analyze the model selection consistency of least absolute shrinkage and selection operator (Lasso), both with and without post-thresholding, for high-dimensional Ising models. For random regular (RR) graphs of size $p$ with regular node degree $d$ and uniform couplings $\theta_0$, it is rigorously proved that Lasso \textit{without post-thresholding} is model selection consistent in the whole paramagnetic phase with the same order of sample complexity $n=\Omega{(d^3\log{p})}$ as that of $\ell_1$-regularized logistic regression ($\ell_1$-LogR). This result is consistent with the conjecture in Meng, Obuchi, and Kabashima 2021 using the non-rigorous replica method from statistical physics and thus complements it with a rigorous proof. For general tree-like graphs, it is demonstrated that the same result as RR graphs can be obtained under mild assumptions of the dependency condition and incoherence condition. Moreover, we provide a rigorous proof of the model selection consistency of Lasso with post-thresholding for general tree-like graphs in the paramagnetic phase without further assumptions on the dependency and incoherence conditions. Experimental results agree well with our theoretical analysis.