Observable asymptotics of regularized Cox regression models with standard Gaussian designs: a statistical mechanics approach

TL;DR

This paper analyzes the asymptotic behavior of regularized Cox regression models with Gaussian covariates using a statistical mechanics approach, introducing a new AMP algorithm and a data-driven method to estimate model parameters.

Contribution

It develops a novel AMP algorithm for Cox models, derives replica equations for large-sample behavior, and proposes a data-based approach to estimate model parameters without solving complex equations.

Findings

The COX-AMP algorithm accurately estimates model parameters.

Replica symmetric equations describe estimator behavior in high dimensions.

The method enables estimation of signal, noise, and generalization error directly from data.

Abstract

We study the asymptotic behaviour of the Regularized Maximum Partial Likelihood Estimator (RMPLE) in the proportional limit, considering an arbitrary convex regularizer and assuming that the covariates $\mathbf{X}_i\in\mathbb{R}^{p}$ follow a multivariate Gaussian law with covariance $\mathbf{I}_p/p$ for each $i=1, \dots, n$. In order to efficiently compute the estimator under investigation, we propose a modified Approximate Message Passing (AMP) algorithm, that we name COX-AMP, and compare its performance with the Coordinate-wise Descent (CD) algorithm, which is taken as reference. By means of the Replica method, we derive a set of six Replica Symmetric (RS) equations that we show to correctly describe the average behaviour of the estimators when the sample size and the number of covariates is large and commensurate. These equations cannot be solved in practice, as the data generating process (that we are trying to estimate) is not known. However, the update equations of COX-AMP suggest the construction of a local field that can in turn be used to accurately estimate all the RS order parameters of the theory \emph{solely from the data}, \emph{without} actually solving the RS equations. We emphasize that this approach can be applied when the estimator is computed via any method and is not restricted to COX-AMP. Once the RS order parameters are estimated, we have access to the amount of signal and noise in the RMPLE, but also its generalization error, directly from the data. Although we focus on the Partial Likelihood objective, we envisage broader application of the methodology proposed here, for instance to GLMs with nuisance parameters, which include some non-proportional hazards models, e.g. Accelerated Failure Time models.