Evaluation of Generalized Degrees of Freedom for Sparse Estimation by Replica Method

TL;DR

This paper introduces a replica method-based approach to evaluate the generalized degrees of freedom in sparse linear regression, linking it to the effective density of non-zero components, and validates it through numerical algorithms.

Contribution

It provides a novel analytical framework for calculating GDF in sparse models using the replica method, applicable without specific regularization forms.

Findings

GDF expressed via saddle point variables in the replica framework

GDF corresponds to the effective density of non-zero components

Results validated by belief propagation algorithm

Abstract

We develop a method to evaluate the generalized degrees of freedom (GDF), which is a key quantity of a model selection criterion, for linear regression with sparse regularization. Using the replica method, GDF is expressed by the variables that characterize the saddle point of the free energy without depending on the form of the regularization. Within the framework of replica symmetric (RS) analysis, GDF is provided with a physical meaning as the effective density of non-zero components. The validity of our method in the RS phase is supported by the consistency of our results with previous mathematical results. The analytical results in the RS phase are numerically achieved by the belief propagation algorithm.