Correction of overfitting bias in regression models

TL;DR

This paper introduces a jackknife-based method to quantify and correct overfitting bias in maximum likelihood regression estimators when the number of covariates is comparable to the number of observations, improving inference accuracy.

Contribution

It develops a new set of non-linear equations for the statistical properties of ML estimators in high-dimensional settings, enabling bias correction without relying on the replica method.

Findings

The equations accurately predict overfitting bias in simulations.

Shrinkage factors effectively remove bias in various regression models.

Method offers transparent bias correction with minimal assumptions.

Abstract

Regression analysis based on many covariates is becoming increasingly common. However, when the number of covariates $p$ is of the same order as the number of observations $n$, maximum likelihood regression becomes unreliable due to overfitting. This typically leads to systematic estimation biases and increased estimator variances. It is crucial for inference and prediction to quantify these effects correctly. Several methods have been proposed in literature to overcome overfitting bias or adjust estimates. The vast majority of these focus on the regression parameters. But failure to estimate correctly also the nuisance parameters may lead to significant errors in confidence statements and outcome prediction. In this paper we present a jacknife method for deriving a compact set of non-linear equations which describe the statistical properties of the ML estimator in the regime where $p=O(n)$ and under the hypothesis of normally distributed covariates. These equations enable one to compute the overfitting bias of maximum likelihood (ML) estimators in parametric regression models as functions of $\zeta = p/n$. We then use these equations to compute shrinkage factors in order to remove the overfitting bias of maximum likelihood (ML) estimators. This new derivation offers various benefits over the replica approach in terms of increased transparency and reduced assumptions. To illustrate the theory we performed simulation studies for multiple regression models. In all cases we find excellent agreement between theory and simulations.