Bounded-memory adjusted scores estimation in generalized linear models with large data sets

TL;DR

This paper introduces two memory-efficient iterative algorithms for estimating generalized linear models with bias-reducing adjusted scores, suitable for large-scale and distributed data sets, using incremental QR decompositions.

Contribution

It develops novel IWLS variants that operate with limited memory and can handle distributed data, advancing scalable bias-reduction in GLMs.

Findings

Algorithms successfully applied to datasets with millions of observations.

Memory requirements are significantly reduced compared to traditional methods.

Methods are adaptable for distributed and privacy-sensitive data environments.

Abstract

The widespread use of maximum Jeffreys'-prior penalized likelihood in binomial-response generalized linear models, and in logistic regression, in particular, are supported by the results of Kosmidis and Firth (2021, Biometrika), who show that the resulting estimates are always finite-valued, even in cases where the maximum likelihood estimates are not, which is a practical issue regardless of the size of the data set. In logistic regression, the implied adjusted score equations are formally bias-reducing in asymptotic frameworks with a fixed number of parameters and appear to deliver a substantial reduction in the persistent bias of the maximum likelihood estimator in high-dimensional settings where the number of parameters grows asymptotically as a proportion of the number of observations. In this work, we develop and present two new variants of iteratively reweighted least squares for estimating generalized linear models with adjusted score equations for mean bias reduction and maximization of the likelihood penalized by a positive power of the Jeffreys-prior penalty, which eliminate the requirement of storing $O(n)$ quantities in memory, and can operate with data sets that exceed computer memory or even hard drive capacity. We achieve that through incremental QR decompositions, which enable IWLS iterations to have access only to data chunks of predetermined size. Both procedures can also be readily adapted to fit generalized linear models when distinct parts of the data is stored across different sites and, due to privacy concerns, cannot be fully transferred across sites. We assess the procedures through a real-data application with millions of observations.