Statistical mechanics of sparse generalization and model selection

TL;DR

This paper uses statistical mechanics to analyze sparse model selection in high-dimensional inference, comparing different dilution methods and highlighting the near-optimal performance of $L_0$ dilution in certain regimes.

Contribution

It provides a theoretical analysis of sparse generalization using replica methods, comparing naive, $L_1$, and $L_0$ dilutions, and identifies conditions where $L_0$ is nearly optimal.

Findings

$L_p$ dilutions outperform naive methods

$L_0$ dilution nearly perfect in some regimes

$L_0$ outperforms $L_1$ in specific conditions

Abstract

One of the crucial tasks in many inference problems is the extraction of sparse information out of a given number of high-dimensional measurements. In machine learning, this is frequently achieved using, as a penality term, the $L_p$ norm of the model parameters, with $p\leq 1$ for efficient dilution. Here we propose a statistical-mechanics analysis of the problem in the setting of perceptron memorization and generalization. Using a replica approach, we are able to evaluate the relative performance of naive dilution (obtained by learning without dilution, following by applying a threshold to the model parameters), $L_1$ dilution (which is frequently used in convex optimization) and $L_0$ dilution (which is optimal but computationally hard to implement). Whereas both $L_p$ diluted approaches clearly outperform the naive approach, we find a small region where $L_0$ works almost perfectly and strongly outperforms the simpler to implement $L_1$ dilution.