Conjugate priors and bias reduction for logistic regression models

TL;DR

This paper introduces a conjugate prior penalty for logistic regression that guarantees finite estimates, approximates Firth's bias reduction method, and improves computational efficiency and inference.

Contribution

It proposes a Bayesian-inspired conjugate prior approach that ensures estimator existence, reduces bias, and enhances scalability in logistic regression models.

Findings

Estimator always exists due to the prior penalty.

The method provides a close approximation to Firth's bias reduction.

Significant computational improvements over existing methods.

Abstract

Logistic regression models for binomial responses are routinely used in statistical practice. However, the maximum likelihood estimate may not exist due to data separability. We address this issue by considering a conjugate prior penalty which always produces finite estimates. Such a specification has a clear Bayesian interpretation and enjoys several invariance properties, making it an appealing prior choice. We show that the proposed method leads to an accurate approximation of the reduced-bias approach of Firth (1993), resulting in estimators with smaller asymptotic bias than the maximum-likelihood and whose existence is always guaranteed. Moreover, the considered penalized likelihood can be expressed as a genuine likelihood, in which the original data are replaced with a collection of pseudo-counts. Hence, our approach may leverage well established and scalable algorithms for logistic regression. We compare our estimator with alternative reduced-bias methods, vastly improving their computational performance and achieving appealing inferential results.