A Fast Divide-and-Conquer Sparse Cox Regression

TL;DR

This paper introduces a fast divide-and-conquer algorithm for sparse Cox regression that efficiently handles massive datasets, maintaining statistical accuracy while significantly reducing computation time.

Contribution

The paper presents a novel DAC algorithm that combines linear approximation and least squares to efficiently fit sparse Cox models on large-scale survival data.

Findings

Outperforms existing methods in computational speed

Achieves similar statistical efficiency as full data methods

Successfully applied to large-scale survival datasets

Abstract

We propose a computationally and statistically efficient divide-and-conquer (DAC) algorithm to fit sparse Cox regression to massive datasets where the sample size $n_0$ is exceedingly large and the covariate dimension $p$ is not small but $n_0\gg p$. The proposed algorithm achieves computational efficiency through a one-step linear approximation followed by a least square approximation to the partial likelihood (PL). These sequences of linearization enable us to maximize the PL with only a small subset and perform penalized estimation via a fast approximation to the PL. The algorithm is applicable for the analysis of both time-independent and time-dependent survival data. Simulations suggest that the proposed DAC algorithm substantially outperforms the full sample-based estimators and the existing DAC algorithm with respect to the computational speed, while it achieves similar statistical efficiency as the full sample-based estimators. The proposed algorithm was applied to an extraordinarily large time-independent survival dataset and an extraordinarily large time-dependent survival dataset for the prediction of heart failure-specific readmission within 30 days among Medicare heart failure patients.