Prediction Errors for Penalized Regressions based on Generalized Approximate Message Passing

TL;DR

This paper analyzes the prediction errors of penalized generalized linear models using GAMP and replica methods, comparing estimators like $C_p$, information criteria, and LOOCV, especially in high-dimensional settings.

Contribution

It introduces a unified framework for estimating prediction errors in penalized GLMs using GAMP, highlighting discrepancies in high-dimensional regimes and connecting information criteria with LOOCV.

Findings

Estimators coincide when model parameters are few.

Discrepancies arise when parameters exceed data dimension.

Information criteria can be expressed via GAMP-derived variance.

Abstract

We discuss the prediction accuracy of assumed statistical models in terms of prediction errors for the generalized linear model and penalized maximum likelihood methods. We derive the forms of estimators for the prediction errors, such as $C_p$ criterion, information criteria, and leave-one-out cross validation (LOOCV) error, using the generalized approximate message passing (GAMP) algorithm and replica method. These estimators coincide with each other when the number of model parameters is sufficiently small; however, there is a discrepancy between them in particular in the parameter region where the number of model parameters is larger than the data dimension. In this paper, we review the prediction errors and corresponding estimators, and discuss their differences. In the framework of GAMP, we show that the information criteria can be expressed by using the variance of the estimates. Further, we demonstrate how to approach LOOCV error from the information criteria by utilizing the expression provided by GAMP.