A modern maximum-likelihood theory for high-dimensional logistic regression

TL;DR

This paper develops a new theoretical framework for high-dimensional logistic regression, revealing that classical inference methods are unreliable when the number of variables grows proportionally with the sample size, and proposes adjustments for accurate inference.

Contribution

It introduces a modern maximum-likelihood theory that accurately predicts bias, variance, and distribution of estimators in high-dimensional logistic regression, improving inference reliability.

Findings

MLE is biased and has greater variability than classical theory predicts.

Likelihood-ratio test does not follow chi-square distribution in high dimensions.

Proposes a procedure to adjust inference based on a single scalar parameter.

Abstract

Every student in statistics or data science learns early on that when the sample size largely exceeds the number of variables, fitting a logistic model produces estimates that are approximately unbiased. Every student also learns that there are formulas to predict the variability of these estimates which are used for the purpose of statistical inference; for instance, to produce p-values for testing the significance of regression coefficients. Although these formulas come from large sample asymptotics, we are often told that we are on reasonably safe grounds when $n$ is large in such a way that $n \ge 5p$ or $n \ge 10p$. This paper shows that this is far from the case, and consequently, inferences routinely produced by common software packages are often unreliable. Consider a logistic model with independent features in which $n$ and $p$ become increasingly large in a fixed ratio. Then we show that (1) the MLE is biased, (2) the variability of the MLE is far greater than classically predicted, and (3) the commonly used likelihood-ratio test (LRT) is not distributed as a chi-square. The bias of the MLE is extremely problematic as it yields completely wrong predictions for the probability of a case based on observed values of the covariates. We develop a new theory, which asymptotically predicts (1) the bias of the MLE, (2) the variability of the MLE, and (3) the distribution of the LRT. We empirically also demonstrate that these predictions are extremely accurate in finite samples. Further, an appealing feature is that these novel predictions depend on the unknown sequence of regression coefficients only through a single scalar, the overall strength of the signal. This suggests very concrete procedures to adjust inference; we describe one such procedure learning a single parameter from data and producing accurate inference