Estimator of Prediction Error Based on Approximate Message Passing for Penalized Linear Regression

TL;DR

This paper introduces an AMP-based estimator for prediction error in penalized linear regression, applicable to various sparse penalties, and demonstrates its asymptotic unbiasedness and effectiveness in model selection.

Contribution

It develops a penalty-independent prediction error estimator using AMP, improving model selection especially with nonconvex penalties.

Findings

Estimator is asymptotically unbiased for Gaussian data.

Performs well with nonconvex sparse penalties.

Model selection aligns closely with true prediction error.

Abstract

We propose an estimator of prediction error using an approximate message passing (AMP) algorithm that can be applied to a broad range of sparse penalties. Following Stein's lemma, the estimator of the generalized degrees of freedom, which is a key quantity for the construction of the estimator of the prediction error, is calculated at the AMP fixed point. The resulting form of the AMP-based estimator does not depend on the penalty function, and its value can be further improved by considering the correlation between predictors. The proposed estimator is asymptotically unbiased when the components of the predictors and response variables are independently generated according to a Gaussian distribution. We examine the behaviour of the estimator for real data under nonconvex sparse penalties, where Akaike's information criterion does not correspond to an unbiased estimator of the prediction error. The model selected by the proposed estimator is close to that which minimizes the true prediction error.