Application of generalized linear models in big data: a divide and recombine (D&R) approach

TL;DR

This paper reviews divide and recombine strategies for fitting generalized linear models to large datasets, proposing a new method for estimating standard errors that achieves efficiency comparable to full data analysis.

Contribution

It introduces a novel sequential partitioning method and a practical approach for estimating standard errors in D&R strategies for GLMs.

Findings

The proposed standard error estimation method is theoretically justified.

D&R estimators with the new standard error are as efficient as full data estimates.

Validation on synthetic data confirms accuracy and consistency with existing R packages.

Abstract

D&R is a statistical approach designed to handle large and complex datasets. It partitions the dataset into several manageable subsets and subsequently applies the analytic method to each subset independently to obtain results. Finally, the results from each subset are combined to yield the results for the entire dataset. D&R strategies can be implemented to fit GLMs to datasets too large for conventional methods. Several D&R strategies are available for different GLMs, some of which are theoretically justified but lack practical validation. A significant limitation is the theoretical and practical justification for estimating combined standard errors and confidence intervals. This paper reviews D&R strategies for GLMs and proposes a method to determine the combined standard error for D&R-based estimators. In addition to the traditional dataset division procedures, we propose a different division method named sequential partitioning for D&R-based estimators on GLMs. We show that the obtained D&R estimator with the proposed standard error attains equivalent efficiency as the full data estimate. We illustrate this on a large synthetic dataset and verify that the results from D&R are accurate and identical to those from other available R packages.