Bayesian Cox Regression for Large-scale Inference with Applications to Electronic Health Records

TL;DR

This paper introduces a Bayesian Cox regression method tailored for large-scale, high-dimensional biomedical datasets, enabling efficient inference and uncertainty estimation in time-to-event analysis.

Contribution

It develops a scalable Bayesian approach using stochastic variational inference for Cox models, suitable for datasets with millions of records and thousands of covariates.

Findings

Effective in large-scale datasets with millions of data points.

Provides reliable uncertainty estimates in high-dimensional settings.

Demonstrated utility on UK Biobank myocardial infarction data.

Abstract

The Cox model is an indispensable tool for time-to-event analysis, particularly in biomedical research. However, medicine is undergoing a profound transformation, generating data at an unprecedented scale, which opens new frontiers to study and understand diseases. With the wealth of data collected, new challenges for statistical inference arise, as datasets are often high dimensional, exhibit an increasing number of measurements at irregularly spaced time points, and are simply too large to fit in memory. Many current implementations for time-to-event analysis are ill-suited for these problems as inference is computationally demanding and requires access to the full data at once. Here we propose a Bayesian version for the counting process representation of Cox's partial likelihood for efficient inference on large-scale datasets with millions of data points and thousands of time-dependent covariates. Through the combination of stochastic variational inference and a reweighting of the log-likelihood, we obtain an approximation for the posterior distribution that factorizes over subsamples of the data, enabling the analysis in big data settings. Crucially, the method produces viable uncertainty estimates for large-scale and high-dimensional datasets. We show the utility of our method through a simulation study and an application to myocardial infarction in the UK Biobank.