Ising Model Selection Using $\ell_{1}$-Regularized Linear Regression: A Statistical Mechanics Analysis

TL;DR

This paper uses statistical mechanics to analyze the sample complexity and performance of $ abla_1$-regularized linear regression for Ising model selection, showing it is consistent with logistic regression in typical cases.

Contribution

It provides a theoretical analysis of $ abla_1$-LinR for Ising model selection, including sample complexity estimates and a method to predict non-asymptotic behavior, applicable to various $ abla_1$-regularized estimators.

Findings

$ abla_1$-LinR is model selection consistent with logistic regression.

Sample complexity is $O(\log N)$ for typical random regular graphs.

Theoretical predictions agree well with simulations even for graphs with many loops.

Abstract

We theoretically analyze the typical learning performance of $\ell_{1}$-regularized linear regression ($\ell_1$-LinR) for Ising model selection using the replica method from statistical mechanics. For typical random regular graphs in the paramagnetic phase, an accurate estimate of the typical sample complexity of $\ell_1$-LinR is obtained. Remarkably, despite the model misspecification, $\ell_1$-LinR is model selection consistent with the same order of sample complexity as $\ell_{1}$-regularized logistic regression ($\ell_1$-LogR), i.e., $M=\mathcal{O}\left(\log N\right)$, where $N$ is the number of variables of the Ising model. Moreover, we provide an efficient method to accurately predict the non-asymptotic behavior of $\ell_1$-LinR for moderate $M, N$, such as precision and recall. Simulations show a fairly good agreement between theoretical predictions and experimental results, even for graphs with many loops, which supports our findings. Although this paper mainly focuses on $\ell_1$-LinR, our method is readily applicable for precisely characterizing the typical learning performances of a wide class of $\ell_{1}$-regularized $M$-estimators including $\ell_1$-LogR and interaction screening.